[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250905 19:57:45.917140 26858 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.26.58.190:39653
I20250905 19:57:45.919608 26858 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 19:57:45.921072 26858 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:57:45.942942 26867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:57:45.942955 26865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:57:45.942942 26864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:57:45.944864 26858 server_base.cc:1047] running on GCE node
I20250905 19:57:47.195617 26858 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:57:47.196069 26858 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:57:47.196437 26858 hybrid_clock.cc:648] HybridClock initialized: now 1757102267196405 us; error 0 us; skew 500 ppm
I20250905 19:57:47.197806 26858 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:57:47.204363 26858 webserver.cc:480] Webserver started at http://127.26.58.190:46859/ using document root <none> and password file <none>
I20250905 19:57:47.205389 26858 fs_manager.cc:362] Metadata directory not provided
I20250905 19:57:47.205642 26858 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:57:47.206135 26858 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:57:47.213119 26858 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1757102265724956-26858-0/minicluster-data/master-0-root/instance:
uuid: "fe6db7b21b684612a333a0c0d71626ef"
format_stamp: "Formatted at 2025-09-05 19:57:47 on dist-test-slave-v26w"
I20250905 19:57:47.222348 26858 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.010s sys 0.000s
I20250905 19:57:47.228781 26873 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:57:47.230373 26858 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20250905 19:57:47.230872 26858 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1757102265724956-26858-0/minicluster-data/master-0-root
uuid: "fe6db7b21b684612a333a0c0d71626ef"
format_stamp: "Formatted at 2025-09-05 19:57:47 on dist-test-slave-v26w"
I20250905 19:57:47.231267 26858 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1757102265724956-26858-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1757102265724956-26858-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1757102265724956-26858-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:57:47.311764 26858 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:57:47.313441 26858 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 19:57:47.313993 26858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:57:47.401473 26858 rpc_server.cc:307] RPC server started. Bound to: 127.26.58.190:39653
I20250905 19:57:47.401540 26924 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.58.190:39653 every 8 connection(s)
I20250905 19:57:47.411098 26925 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:57:47.438360 26925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe6db7b21b684612a333a0c0d71626ef: Bootstrap starting.
I20250905 19:57:47.445032 26925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe6db7b21b684612a333a0c0d71626ef: Neither blocks nor log segments found. Creating new log.
I20250905 19:57:47.447384 26925 log.cc:826] T 00000000000000000000000000000000 P fe6db7b21b684612a333a0c0d71626ef: Log is configured to *not* fsync() on all Append() calls
I20250905 19:57:47.452895 26925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe6db7b21b684612a333a0c0d71626ef: No bootstrap required, opened a new log
I20250905 19:57:47.478803 26925 raft_consensus.cc:357] T 00000000000000000000000000000000 P fe6db7b21b684612a333a0c0d71626ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6db7b21b684612a333a0c0d71626ef" member_type: VOTER }
I20250905 19:57:47.479460 26925 raft_consensus.cc:383] T 00000000000000000000000000000000 P fe6db7b21b684612a333a0c0d71626ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:57:47.479689 26925 raft_consensus.cc:738] T 00000000000000000000000000000000 P fe6db7b21b684612a333a0c0d71626ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe6db7b21b684612a333a0c0d71626ef, State: Initialized, Role: FOLLOWER
I20250905 19:57:47.480581 26925 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe6db7b21b684612a333a0c0d71626ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6db7b21b684612a333a0c0d71626ef" member_type: VOTER }
I20250905 19:57:47.481106 26925 raft_consensus.cc:397] T 00000000000000000000000000000000 P fe6db7b21b684612a333a0c0d71626ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 19:57:47.481353 26925 raft_consensus.cc:491] T 00000000000000000000000000000000 P fe6db7b21b684612a333a0c0d71626ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 19:57:47.481658 26925 raft_consensus.cc:3058] T 00000000000000000000000000000000 P fe6db7b21b684612a333a0c0d71626ef [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:57:47.486086 26925 raft_consensus.cc:513] T 00000000000000000000000000000000 P fe6db7b21b684612a333a0c0d71626ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6db7b21b684612a333a0c0d71626ef" member_type: VOTER }
I20250905 19:57:47.486925 26925 leader_election.cc:304] T 00000000000000000000000000000000 P fe6db7b21b684612a333a0c0d71626ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe6db7b21b684612a333a0c0d71626ef; no voters:
I20250905 19:57:47.489190 26925 leader_election.cc:290] T 00000000000000000000000000000000 P fe6db7b21b684612a333a0c0d71626ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 19:57:47.489609 26928 raft_consensus.cc:2802] T 00000000000000000000000000000000 P fe6db7b21b684612a333a0c0d71626ef [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:57:47.492908 26928 raft_consensus.cc:695] T 00000000000000000000000000000000 P fe6db7b21b684612a333a0c0d71626ef [term 1 LEADER]: Becoming Leader. State: Replica: fe6db7b21b684612a333a0c0d71626ef, State: Running, Role: LEADER
I20250905 19:57:47.494134 26928 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe6db7b21b684612a333a0c0d71626ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6db7b21b684612a333a0c0d71626ef" member_type: VOTER }
I20250905 19:57:47.496400 26925 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe6db7b21b684612a333a0c0d71626ef [sys.catalog]: configured and running, proceeding with master startup.
I20250905 19:57:47.507777 26930 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe6db7b21b684612a333a0c0d71626ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe6db7b21b684612a333a0c0d71626ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6db7b21b684612a333a0c0d71626ef" member_type: VOTER } }
I20250905 19:57:47.507607 26929 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe6db7b21b684612a333a0c0d71626ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe6db7b21b684612a333a0c0d71626ef. Latest consensus state: current_term: 1 leader_uuid: "fe6db7b21b684612a333a0c0d71626ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6db7b21b684612a333a0c0d71626ef" member_type: VOTER } }
I20250905 19:57:47.508530 26930 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe6db7b21b684612a333a0c0d71626ef [sys.catalog]: This master's current role is: LEADER
I20250905 19:57:47.508615 26929 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe6db7b21b684612a333a0c0d71626ef [sys.catalog]: This master's current role is: LEADER
I20250905 19:57:47.514848 26937 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 19:57:47.533983 26937 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 19:57:47.538527 26858 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250905 19:57:47.559504 26937 catalog_manager.cc:1349] Generated new cluster ID: 66daf4ec084e4e4cb16a5026c2a45ab4
I20250905 19:57:47.559832 26937 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 19:57:47.579147 26937 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 19:57:47.581171 26937 catalog_manager.cc:1506] Loading token signing keys...
I20250905 19:57:47.597744 26937 catalog_manager.cc:5955] T 00000000000000000000000000000000 P fe6db7b21b684612a333a0c0d71626ef: Generated new TSK 0
I20250905 19:57:47.598845 26937 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 19:57:47.606146 26858 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:57:47.615552 26947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:57:47.617691 26948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:57:47.622289 26858 server_base.cc:1047] running on GCE node
W20250905 19:57:47.623302 26950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:57:47.625092 26858 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:57:47.625451 26858 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:57:47.625707 26858 hybrid_clock.cc:648] HybridClock initialized: now 1757102267625679 us; error 0 us; skew 500 ppm
I20250905 19:57:47.626673 26858 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:57:47.631994 26858 webserver.cc:480] Webserver started at http://127.26.58.129:42271/ using document root <none> and password file <none>
I20250905 19:57:47.632880 26858 fs_manager.cc:362] Metadata directory not provided
I20250905 19:57:47.633188 26858 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:57:47.633594 26858 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:57:47.635470 26858 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1757102265724956-26858-0/minicluster-data/ts-0-root/instance:
uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
format_stamp: "Formatted at 2025-09-05 19:57:47 on dist-test-slave-v26w"
I20250905 19:57:47.645251 26858 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.001s sys 0.009s
I20250905 19:57:47.651512 26955 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:57:47.652626 26858 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250905 19:57:47.653044 26858 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1757102265724956-26858-0/minicluster-data/ts-0-root
uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
format_stamp: "Formatted at 2025-09-05 19:57:47 on dist-test-slave-v26w"
I20250905 19:57:47.653458 26858 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1757102265724956-26858-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1757102265724956-26858-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1757102265724956-26858-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:57:47.679971 26858 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:57:47.682255 26858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:57:47.685554 26858 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:57:47.691967 26858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:57:47.692401 26858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:57:47.692768 26858 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:57:47.693024 26858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:57:47.810639 26858 rpc_server.cc:307] RPC server started. Bound to: 127.26.58.129:41107
I20250905 19:57:47.810712 27017 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.58.129:41107 every 8 connection(s)
I20250905 19:57:47.819393 26858 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:57:47.832151 27022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:57:47.835448 27023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:57:47.846817 26858 server_base.cc:1047] running on GCE node
W20250905 19:57:47.849349 27026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:57:47.850632 26858 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:57:47.850937 26858 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:57:47.851261 26858 hybrid_clock.cc:648] HybridClock initialized: now 1757102267851238 us; error 0 us; skew 500 ppm
I20250905 19:57:47.851778 27018 heartbeater.cc:344] Connected to a master server at 127.26.58.190:39653
I20250905 19:57:47.852223 26858 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:57:47.852349 27018 heartbeater.cc:461] Registering TS with master...
I20250905 19:57:47.853897 27018 heartbeater.cc:507] Master 127.26.58.190:39653 requested a full tablet report, sending...
I20250905 19:57:47.856702 26858 webserver.cc:480] Webserver started at http://127.26.58.130:43449/ using document root <none> and password file <none>
I20250905 19:57:47.857491 26858 fs_manager.cc:362] Metadata directory not provided
I20250905 19:57:47.857761 26858 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:57:47.858155 26858 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:57:47.858198 26890 ts_manager.cc:194] Registered new tserver with Master: ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107)
I20250905 19:57:47.859923 26858 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1757102265724956-26858-0/minicluster-data/ts-1-root/instance:
uuid: "a5dbf236134d43158377e0310cee23d7"
format_stamp: "Formatted at 2025-09-05 19:57:47 on dist-test-slave-v26w"
I20250905 19:57:47.861865 26890 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35910
I20250905 19:57:47.868330 26858 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.003s
I20250905 19:57:47.873312 27030 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:57:47.874269 26858 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250905 19:57:47.874768 26858 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1757102265724956-26858-0/minicluster-data/ts-1-root
uuid: "a5dbf236134d43158377e0310cee23d7"
format_stamp: "Formatted at 2025-09-05 19:57:47 on dist-test-slave-v26w"
I20250905 19:57:47.875233 26858 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1757102265724956-26858-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1757102265724956-26858-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1757102265724956-26858-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:57:47.893393 26858 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:57:47.895315 26858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:57:47.898116 26858 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:57:47.902033 26858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:57:47.902351 26858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:57:47.902683 26858 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:57:47.902894 26858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:57:48.003535 26858 rpc_server.cc:307] RPC server started. Bound to: 127.26.58.130:44875
I20250905 19:57:48.003708 27092 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.58.130:44875 every 8 connection(s)
I20250905 19:57:48.011111 26858 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:57:48.021232 27096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:57:48.026983 27097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:57:48.029073 27093 heartbeater.cc:344] Connected to a master server at 127.26.58.190:39653
I20250905 19:57:48.029492 27093 heartbeater.cc:461] Registering TS with master...
I20250905 19:57:48.030500 27093 heartbeater.cc:507] Master 127.26.58.190:39653 requested a full tablet report, sending...
I20250905 19:57:48.032714 26858 server_base.cc:1047] running on GCE node
I20250905 19:57:48.032960 26890 ts_manager.cc:194] Registered new tserver with Master: a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875)
I20250905 19:57:48.034619 26890 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35924
W20250905 19:57:48.037035 27099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:57:48.038002 26858 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:57:48.038206 26858 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:57:48.038379 26858 hybrid_clock.cc:648] HybridClock initialized: now 1757102268038362 us; error 0 us; skew 500 ppm
I20250905 19:57:48.038954 26858 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:57:48.041664 26858 webserver.cc:480] Webserver started at http://127.26.58.131:34453/ using document root <none> and password file <none>
I20250905 19:57:48.042173 26858 fs_manager.cc:362] Metadata directory not provided
I20250905 19:57:48.042354 26858 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:57:48.042670 26858 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:57:48.043723 26858 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1757102265724956-26858-0/minicluster-data/ts-2-root/instance:
uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
format_stamp: "Formatted at 2025-09-05 19:57:48 on dist-test-slave-v26w"
I20250905 19:57:48.048466 26858 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250905 19:57:48.052201 27104 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:57:48.053221 26858 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250905 19:57:48.053534 26858 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1757102265724956-26858-0/minicluster-data/ts-2-root
uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
format_stamp: "Formatted at 2025-09-05 19:57:48 on dist-test-slave-v26w"
I20250905 19:57:48.053835 26858 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1757102265724956-26858-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1757102265724956-26858-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1757102265724956-26858-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:57:48.070923 26858 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:57:48.072299 26858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:57:48.074995 26858 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:57:48.078814 26858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:57:48.079021 26858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:57:48.079303 26858 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:57:48.079447 26858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:57:48.190217 26858 rpc_server.cc:307] RPC server started. Bound to: 127.26.58.131:35561
I20250905 19:57:48.190385 27166 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.58.131:35561 every 8 connection(s)
I20250905 19:57:48.208411 27167 heartbeater.cc:344] Connected to a master server at 127.26.58.190:39653
I20250905 19:57:48.208853 27167 heartbeater.cc:461] Registering TS with master...
I20250905 19:57:48.209619 27167 heartbeater.cc:507] Master 127.26.58.190:39653 requested a full tablet report, sending...
I20250905 19:57:48.211843 26890 ts_manager.cc:194] Registered new tserver with Master: ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:57:48.212145 26858 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016734243s
I20250905 19:57:48.213788 26890 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35934
I20250905 19:57:48.216893 26858 test_util.cc:276] Using random seed: 1082652194
I20250905 19:57:48.270669 26890 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35940:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:57:48.274606 26890 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:57:48.414719 26980 tablet_service.cc:1468] Processing CreateTablet for tablet 2f9344374d5649489c2f4336e9a1b15b (DEFAULT_TABLE table=test-workload [id=0c691029e9b24335b3938d4610e5d99b]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250905 19:57:48.413820 26981 tablet_service.cc:1468] Processing CreateTablet for tablet c6a00cc443b94edcb46fbfdf1d7ac32c (DEFAULT_TABLE table=test-workload [id=0c691029e9b24335b3938d4610e5d99b]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250905 19:57:48.414987 26983 tablet_service.cc:1468] Processing CreateTablet for tablet c08349e8a8394ee182d5d0928074181f (DEFAULT_TABLE table=test-workload [id=0c691029e9b24335b3938d4610e5d99b]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250905 19:57:48.420006 26980 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f9344374d5649489c2f4336e9a1b15b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:57:48.413434 26982 tablet_service.cc:1468] Processing CreateTablet for tablet 7640f63d346f4d078ecb8d0e8a6c5670 (DEFAULT_TABLE table=test-workload [id=0c691029e9b24335b3938d4610e5d99b]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250905 19:57:48.425345 26981 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a00cc443b94edcb46fbfdf1d7ac32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:57:48.423107 26978 tablet_service.cc:1468] Processing CreateTablet for tablet b2fabc1ff8894320aa02e55c92ecfb1e (DEFAULT_TABLE table=test-workload [id=0c691029e9b24335b3938d4610e5d99b]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250905 19:57:48.429086 26983 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c08349e8a8394ee182d5d0928074181f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:57:48.427453 26979 tablet_service.cc:1468] Processing CreateTablet for tablet 71b161547e6a40e7b93b7f852b76e7d0 (DEFAULT_TABLE table=test-workload [id=0c691029e9b24335b3938d4610e5d99b]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250905 19:57:48.431509 26975 tablet_service.cc:1468] Processing CreateTablet for tablet f6cfad67eeab45ddb61ef84d09f09b6b (DEFAULT_TABLE table=test-workload [id=0c691029e9b24335b3938d4610e5d99b]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250905 19:57:48.428015 26977 tablet_service.cc:1468] Processing CreateTablet for tablet dbaec702dd3f48f2a1fcf12f8befed7e (DEFAULT_TABLE table=test-workload [id=0c691029e9b24335b3938d4610e5d99b]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20250905 19:57:48.431618 26976 tablet_service.cc:1468] Processing CreateTablet for tablet 685a60cc8a6841c6808fcba893636187 (DEFAULT_TABLE table=test-workload [id=0c691029e9b24335b3938d4610e5d99b]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250905 19:57:48.435196 26982 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7640f63d346f4d078ecb8d0e8a6c5670. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:57:48.441213 27057 tablet_service.cc:1468] Processing CreateTablet for tablet 7640f63d346f4d078ecb8d0e8a6c5670 (DEFAULT_TABLE table=test-workload [id=0c691029e9b24335b3938d4610e5d99b]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250905 19:57:48.441820 26979 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71b161547e6a40e7b93b7f852b76e7d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:57:48.438102 27058 tablet_service.cc:1468] Processing CreateTablet for tablet c08349e8a8394ee182d5d0928074181f (DEFAULT_TABLE table=test-workload [id=0c691029e9b24335b3938d4610e5d99b]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250905 19:57:48.442853 27056 tablet_service.cc:1468] Processing CreateTablet for tablet c6a00cc443b94edcb46fbfdf1d7ac32c (DEFAULT_TABLE table=test-workload [id=0c691029e9b24335b3938d4610e5d99b]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250905 19:57:48.445500 27055 tablet_service.cc:1468] Processing CreateTablet for tablet 2f9344374d5649489c2f4336e9a1b15b (DEFAULT_TABLE table=test-workload [id=0c691029e9b24335b3938d4610e5d99b]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250905 19:57:48.451460 26976 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 685a60cc8a6841c6808fcba893636187. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:57:48.452056 27053 tablet_service.cc:1468] Processing CreateTablet for tablet b2fabc1ff8894320aa02e55c92ecfb1e (DEFAULT_TABLE table=test-workload [id=0c691029e9b24335b3938d4610e5d99b]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250905 19:57:48.449294 27054 tablet_service.cc:1468] Processing CreateTablet for tablet 71b161547e6a40e7b93b7f852b76e7d0 (DEFAULT_TABLE table=test-workload [id=0c691029e9b24335b3938d4610e5d99b]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250905 19:57:48.452188 27057 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7640f63d346f4d078ecb8d0e8a6c5670. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:57:48.460134 27052 tablet_service.cc:1468] Processing CreateTablet for tablet dbaec702dd3f48f2a1fcf12f8befed7e (DEFAULT_TABLE table=test-workload [id=0c691029e9b24335b3938d4610e5d99b]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20250905 19:57:48.460103 26975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6cfad67eeab45ddb61ef84d09f09b6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:57:48.465154 27051 tablet_service.cc:1468] Processing CreateTablet for tablet 685a60cc8a6841c6808fcba893636187 (DEFAULT_TABLE table=test-workload [id=0c691029e9b24335b3938d4610e5d99b]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250905 19:57:48.465998 27055 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f9344374d5649489c2f4336e9a1b15b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:57:48.468140 27050 tablet_service.cc:1468] Processing CreateTablet for tablet f6cfad67eeab45ddb61ef84d09f09b6b (DEFAULT_TABLE table=test-workload [id=0c691029e9b24335b3938d4610e5d99b]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250905 19:57:48.469159 26978 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2fabc1ff8894320aa02e55c92ecfb1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:57:48.470348 27052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbaec702dd3f48f2a1fcf12f8befed7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:57:48.471576 27053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2fabc1ff8894320aa02e55c92ecfb1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:57:48.476723 27056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a00cc443b94edcb46fbfdf1d7ac32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:57:48.481783 27058 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c08349e8a8394ee182d5d0928074181f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:57:48.479357 26977 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbaec702dd3f48f2a1fcf12f8befed7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:57:48.483875 27050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6cfad67eeab45ddb61ef84d09f09b6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:57:48.500032 27051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 685a60cc8a6841c6808fcba893636187. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:57:48.514000 27054 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71b161547e6a40e7b93b7f852b76e7d0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:57:48.539125 26938 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250905 19:57:48.610983 27132 tablet_service.cc:1468] Processing CreateTablet for tablet c08349e8a8394ee182d5d0928074181f (DEFAULT_TABLE table=test-workload [id=0c691029e9b24335b3938d4610e5d99b]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250905 19:57:48.612507 27132 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c08349e8a8394ee182d5d0928074181f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:57:48.631049 27130 tablet_service.cc:1468] Processing CreateTablet for tablet c6a00cc443b94edcb46fbfdf1d7ac32c (DEFAULT_TABLE table=test-workload [id=0c691029e9b24335b3938d4610e5d99b]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250905 19:57:48.637399 27129 tablet_service.cc:1468] Processing CreateTablet for tablet 2f9344374d5649489c2f4336e9a1b15b (DEFAULT_TABLE table=test-workload [id=0c691029e9b24335b3938d4610e5d99b]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250905 19:57:48.640833 27128 tablet_service.cc:1468] Processing CreateTablet for tablet 71b161547e6a40e7b93b7f852b76e7d0 (DEFAULT_TABLE table=test-workload [id=0c691029e9b24335b3938d4610e5d99b]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250905 19:57:48.644129 27130 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a00cc443b94edcb46fbfdf1d7ac32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:57:48.634285 27131 tablet_service.cc:1468] Processing CreateTablet for tablet 7640f63d346f4d078ecb8d0e8a6c5670 (DEFAULT_TABLE table=test-workload [id=0c691029e9b24335b3938d4610e5d99b]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250905 19:57:48.654101 27127 tablet_service.cc:1468] Processing CreateTablet for tablet b2fabc1ff8894320aa02e55c92ecfb1e (DEFAULT_TABLE table=test-workload [id=0c691029e9b24335b3938d4610e5d99b]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250905 19:57:48.663116 27126 tablet_service.cc:1468] Processing CreateTablet for tablet dbaec702dd3f48f2a1fcf12f8befed7e (DEFAULT_TABLE table=test-workload [id=0c691029e9b24335b3938d4610e5d99b]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20250905 19:57:48.661950 27129 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f9344374d5649489c2f4336e9a1b15b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:57:48.668004 27131 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7640f63d346f4d078ecb8d0e8a6c5670. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:57:48.663177 27125 tablet_service.cc:1468] Processing CreateTablet for tablet 685a60cc8a6841c6808fcba893636187 (DEFAULT_TABLE table=test-workload [id=0c691029e9b24335b3938d4610e5d99b]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250905 19:57:48.660001 27124 tablet_service.cc:1468] Processing CreateTablet for tablet f6cfad67eeab45ddb61ef84d09f09b6b (DEFAULT_TABLE table=test-workload [id=0c691029e9b24335b3938d4610e5d99b]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250905 19:57:48.680958 27127 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2fabc1ff8894320aa02e55c92ecfb1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:57:48.683429 27185 tablet_bootstrap.cc:492] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b: Bootstrap starting.
I20250905 19:57:48.690135 27186 tablet_bootstrap.cc:492] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7: Bootstrap starting.
I20250905 19:57:48.696287 27126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbaec702dd3f48f2a1fcf12f8befed7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:57:48.700258 27128 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71b161547e6a40e7b93b7f852b76e7d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:57:48.731810 27125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 685a60cc8a6841c6808fcba893636187. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:57:48.761265 27124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6cfad67eeab45ddb61ef84d09f09b6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:57:48.766188 27185 tablet_bootstrap.cc:654] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b: Neither blocks nor log segments found. Creating new log.
I20250905 19:57:48.802022 27185 tablet_bootstrap.cc:492] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b: No bootstrap required, opened a new log
I20250905 19:57:48.802498 27186 tablet_bootstrap.cc:654] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7: Neither blocks nor log segments found. Creating new log.
I20250905 19:57:48.804291 27185 ts_tablet_manager.cc:1397] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b: Time spent bootstrapping tablet: real 0.176s user 0.012s sys 0.025s
I20250905 19:57:48.809275 27188 tablet_bootstrap.cc:492] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8: Bootstrap starting.
I20250905 19:57:48.814328 27185 raft_consensus.cc:357] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:48.815418 27185 raft_consensus.cc:383] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:57:48.815735 27185 raft_consensus.cc:738] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab704ad8154f4a4590e214cf9c1d7c4b, State: Initialized, Role: FOLLOWER
I20250905 19:57:48.822813 27185 consensus_queue.cc:260] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:48.841877 27186 tablet_bootstrap.cc:492] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7: No bootstrap required, opened a new log
I20250905 19:57:48.845695 27186 ts_tablet_manager.cc:1397] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7: Time spent bootstrapping tablet: real 0.208s user 0.013s sys 0.022s
I20250905 19:57:48.846621 27018 heartbeater.cc:499] Master 127.26.58.190:39653 was elected leader, sending a full tablet report...
I20250905 19:57:48.847127 27188 tablet_bootstrap.cc:654] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8: Neither blocks nor log segments found. Creating new log.
I20250905 19:57:48.850243 27186 raft_consensus.cc:357] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:48.851344 27186 raft_consensus.cc:383] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:57:48.851694 27185 ts_tablet_manager.cc:1428] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b: Time spent starting tablet: real 0.042s user 0.010s sys 0.005s
I20250905 19:57:48.851902 27186 raft_consensus.cc:738] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5dbf236134d43158377e0310cee23d7, State: Initialized, Role: FOLLOWER
I20250905 19:57:48.852771 27185 tablet_bootstrap.cc:492] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b: Bootstrap starting.
I20250905 19:57:48.853055 27186 consensus_queue.cc:260] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:48.870034 27188 tablet_bootstrap.cc:492] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8: No bootstrap required, opened a new log
I20250905 19:57:48.870827 27188 ts_tablet_manager.cc:1397] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8: Time spent bootstrapping tablet: real 0.070s user 0.012s sys 0.021s
I20250905 19:57:48.874889 27188 raft_consensus.cc:357] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:48.875970 27188 raft_consensus.cc:383] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:57:48.876384 27188 raft_consensus.cc:738] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac86c30e5d044b9caf64b0c44f5cf7c8, State: Initialized, Role: FOLLOWER
I20250905 19:57:48.877342 27188 consensus_queue.cc:260] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:48.878587 27185 tablet_bootstrap.cc:654] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b: Neither blocks nor log segments found. Creating new log.
I20250905 19:57:48.890415 27167 heartbeater.cc:499] Master 127.26.58.190:39653 was elected leader, sending a full tablet report...
I20250905 19:57:48.891328 27188 ts_tablet_manager.cc:1428] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8: Time spent starting tablet: real 0.020s user 0.010s sys 0.010s
I20250905 19:57:48.892748 27188 tablet_bootstrap.cc:492] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8: Bootstrap starting.
I20250905 19:57:48.896764 27093 heartbeater.cc:499] Master 127.26.58.190:39653 was elected leader, sending a full tablet report...
I20250905 19:57:48.900003 27186 ts_tablet_manager.cc:1428] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7: Time spent starting tablet: real 0.053s user 0.007s sys 0.035s
I20250905 19:57:48.901403 27186 tablet_bootstrap.cc:492] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7: Bootstrap starting.
I20250905 19:57:48.908133 27186 tablet_bootstrap.cc:654] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7: Neither blocks nor log segments found. Creating new log.
I20250905 19:57:48.908138 27188 tablet_bootstrap.cc:654] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8: Neither blocks nor log segments found. Creating new log.
I20250905 19:57:48.920946 27185 tablet_bootstrap.cc:492] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b: No bootstrap required, opened a new log
I20250905 19:57:48.921217 27188 tablet_bootstrap.cc:492] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8: No bootstrap required, opened a new log
I20250905 19:57:48.921485 27185 ts_tablet_manager.cc:1397] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b: Time spent bootstrapping tablet: real 0.069s user 0.009s sys 0.004s
I20250905 19:57:48.921623 27188 ts_tablet_manager.cc:1397] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.010s
I20250905 19:57:48.921877 27186 tablet_bootstrap.cc:492] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7: No bootstrap required, opened a new log
I20250905 19:57:48.922364 27186 ts_tablet_manager.cc:1397] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.005s
I20250905 19:57:48.932308 27188 raft_consensus.cc:357] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:48.933197 27188 raft_consensus.cc:383] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:57:48.933539 27188 raft_consensus.cc:738] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac86c30e5d044b9caf64b0c44f5cf7c8, State: Initialized, Role: FOLLOWER
I20250905 19:57:48.933317 27186 raft_consensus.cc:357] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:48.932536 27185 raft_consensus.cc:357] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:48.934134 27186 raft_consensus.cc:383] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:57:48.934514 27186 raft_consensus.cc:738] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5dbf236134d43158377e0310cee23d7, State: Initialized, Role: FOLLOWER
I20250905 19:57:48.934350 27188 consensus_queue.cc:260] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:48.937222 27188 ts_tablet_manager.cc:1428] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20250905 19:57:48.938237 27188 tablet_bootstrap.cc:492] T 71b161547e6a40e7b93b7f852b76e7d0 P ac86c30e5d044b9caf64b0c44f5cf7c8: Bootstrap starting.
I20250905 19:57:48.935379 27186 consensus_queue.cc:260] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:48.940667 27186 ts_tablet_manager.cc:1428] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7: Time spent starting tablet: real 0.018s user 0.012s sys 0.002s
I20250905 19:57:48.941704 27186 tablet_bootstrap.cc:492] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7: Bootstrap starting.
I20250905 19:57:48.942723 27193 raft_consensus.cc:491] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:57:48.943235 27193 raft_consensus.cc:513] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:48.948325 27186 tablet_bootstrap.cc:654] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7: Neither blocks nor log segments found. Creating new log.
W20250905 19:57:48.950419 27168 tablet.cc:2378] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 19:57:48.961642 27192 raft_consensus.cc:491] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:57:48.934350 27185 raft_consensus.cc:383] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:57:48.963881 27192 raft_consensus.cc:513] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:48.956991 27193 leader_election.cc:290] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107), a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875)
I20250905 19:57:48.963521 27185 raft_consensus.cc:738] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab704ad8154f4a4590e214cf9c1d7c4b, State: Initialized, Role: FOLLOWER
I20250905 19:57:48.978893 27190 raft_consensus.cc:491] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:57:48.982172 27190 raft_consensus.cc:513] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:48.984896 27185 consensus_queue.cc:260] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:48.989430 27190 leader_election.cc:290] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:57:48.996767 27192 leader_election.cc:290] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:57:48.997407 27188 tablet_bootstrap.cc:654] T 71b161547e6a40e7b93b7f852b76e7d0 P ac86c30e5d044b9caf64b0c44f5cf7c8: Neither blocks nor log segments found. Creating new log.
I20250905 19:57:49.009912 27186 tablet_bootstrap.cc:492] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7: No bootstrap required, opened a new log
I20250905 19:57:49.010772 27186 ts_tablet_manager.cc:1397] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7: Time spent bootstrapping tablet: real 0.069s user 0.017s sys 0.017s
I20250905 19:57:49.025296 27186 raft_consensus.cc:357] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.026082 27186 raft_consensus.cc:383] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:57:49.026377 27186 raft_consensus.cc:738] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5dbf236134d43158377e0310cee23d7, State: Initialized, Role: FOLLOWER
I20250905 19:57:49.027458 27186 consensus_queue.cc:260] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.032312 27186 ts_tablet_manager.cc:1428] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20250905 19:57:49.033393 27193 raft_consensus.cc:491] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:57:49.034083 27193 raft_consensus.cc:513] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.036366 27186 tablet_bootstrap.cc:492] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7: Bootstrap starting.
I20250905 19:57:49.042127 27190 raft_consensus.cc:491] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:57:49.043488 27190 raft_consensus.cc:513] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.038821 27193 leader_election.cc:290] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875), ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107)
I20250905 19:57:49.057637 27190 leader_election.cc:290] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:57:49.082191 26992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c08349e8a8394ee182d5d0928074181f" candidate_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" is_pre_election: true
I20250905 19:57:49.081525 26993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2fabc1ff8894320aa02e55c92ecfb1e" candidate_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" is_pre_election: true
I20250905 19:57:49.093961 27188 tablet_bootstrap.cc:492] T 71b161547e6a40e7b93b7f852b76e7d0 P ac86c30e5d044b9caf64b0c44f5cf7c8: No bootstrap required, opened a new log
I20250905 19:57:49.095535 27185 ts_tablet_manager.cc:1428] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b: Time spent starting tablet: real 0.174s user 0.020s sys 0.006s
I20250905 19:57:49.098114 27188 ts_tablet_manager.cc:1397] T 71b161547e6a40e7b93b7f852b76e7d0 P ac86c30e5d044b9caf64b0c44f5cf7c8: Time spent bootstrapping tablet: real 0.160s user 0.014s sys 0.038s
W20250905 19:57:49.100780 27107 leader_election.cc:343] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:57:49.101770 27107 leader_election.cc:343] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:57:49.103667 27188 raft_consensus.cc:357] T 71b161547e6a40e7b93b7f852b76e7d0 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.109429 27188 raft_consensus.cc:383] T 71b161547e6a40e7b93b7f852b76e7d0 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:57:49.109766 27188 raft_consensus.cc:738] T 71b161547e6a40e7b93b7f852b76e7d0 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac86c30e5d044b9caf64b0c44f5cf7c8, State: Initialized, Role: FOLLOWER
I20250905 19:57:49.111888 27185 tablet_bootstrap.cc:492] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b: Bootstrap starting.
I20250905 19:57:49.125798 27068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7640f63d346f4d078ecb8d0e8a6c5670" candidate_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5dbf236134d43158377e0310cee23d7" is_pre_election: true
I20250905 19:57:49.117990 27188 consensus_queue.cc:260] T 71b161547e6a40e7b93b7f852b76e7d0 P ac86c30e5d044b9caf64b0c44f5cf7c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.133175 27067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "685a60cc8a6841c6808fcba893636187" candidate_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5dbf236134d43158377e0310cee23d7" is_pre_election: true
W20250905 19:57:49.135954 26957 leader_election.cc:343] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:57:49.137516 27067 raft_consensus.cc:2466] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab704ad8154f4a4590e214cf9c1d7c4b in term 0.
I20250905 19:57:49.139701 27188 ts_tablet_manager.cc:1428] T 71b161547e6a40e7b93b7f852b76e7d0 P ac86c30e5d044b9caf64b0c44f5cf7c8: Time spent starting tablet: real 0.040s user 0.005s sys 0.005s
I20250905 19:57:49.143155 26957 leader_election.cc:304] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5dbf236134d43158377e0310cee23d7, ab704ad8154f4a4590e214cf9c1d7c4b; no voters:
I20250905 19:57:49.143580 27186 tablet_bootstrap.cc:654] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7: Neither blocks nor log segments found. Creating new log.
I20250905 19:57:49.145690 27142 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7640f63d346f4d078ecb8d0e8a6c5670" candidate_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" is_pre_election: true
W20250905 19:57:49.149020 26958 leader_election.cc:343] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:57:49.149149 27142 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "685a60cc8a6841c6808fcba893636187" candidate_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" is_pre_election: true
I20250905 19:57:49.149654 26958 leader_election.cc:304] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab704ad8154f4a4590e214cf9c1d7c4b; no voters: a5dbf236134d43158377e0310cee23d7, ac86c30e5d044b9caf64b0c44f5cf7c8
I20250905 19:57:49.150852 27204 raft_consensus.cc:2802] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:57:49.151327 27204 raft_consensus.cc:491] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:57:49.151657 27204 raft_consensus.cc:3058] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:57:49.151932 27188 tablet_bootstrap.cc:492] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8: Bootstrap starting.
I20250905 19:57:49.153182 27190 raft_consensus.cc:2747] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:57:49.154541 26958 leader_election.cc:343] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:57:49.170858 27185 tablet_bootstrap.cc:654] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b: Neither blocks nor log segments found. Creating new log.
I20250905 19:57:49.172269 27192 raft_consensus.cc:491] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:57:49.172775 27192 raft_consensus.cc:513] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.174736 27192 leader_election.cc:290] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:57:49.178182 26992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "685a60cc8a6841c6808fcba893636187" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" is_pre_election: true
I20250905 19:57:49.180150 26993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbaec702dd3f48f2a1fcf12f8befed7e" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" is_pre_election: true
I20250905 19:57:49.183058 27067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2fabc1ff8894320aa02e55c92ecfb1e" candidate_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5dbf236134d43158377e0310cee23d7" is_pre_election: true
W20250905 19:57:49.183516 27033 leader_election.cc:343] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:57:49.187471 27106 leader_election.cc:343] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:57:49.188555 27106 leader_election.cc:304] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac86c30e5d044b9caf64b0c44f5cf7c8; no voters: a5dbf236134d43158377e0310cee23d7, ab704ad8154f4a4590e214cf9c1d7c4b
I20250905 19:57:49.189739 27193 raft_consensus.cc:2747] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:57:49.186918 27068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c08349e8a8394ee182d5d0928074181f" candidate_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5dbf236134d43158377e0310cee23d7" is_pre_election: true
I20250905 19:57:49.191732 27186 tablet_bootstrap.cc:492] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7: No bootstrap required, opened a new log
W20250905 19:57:49.192691 27106 leader_election.cc:343] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:57:49.192891 27186 ts_tablet_manager.cc:1397] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7: Time spent bootstrapping tablet: real 0.157s user 0.015s sys 0.003s
I20250905 19:57:49.193195 27106 leader_election.cc:304] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac86c30e5d044b9caf64b0c44f5cf7c8; no voters: a5dbf236134d43158377e0310cee23d7, ab704ad8154f4a4590e214cf9c1d7c4b
I20250905 19:57:49.194334 27142 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "685a60cc8a6841c6808fcba893636187" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" is_pre_election: true
I20250905 19:57:49.195346 27193 raft_consensus.cc:2747] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:57:49.195853 27141 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbaec702dd3f48f2a1fcf12f8befed7e" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" is_pre_election: true
W20250905 19:57:49.196597 27033 leader_election.cc:343] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:57:49.187860 27204 raft_consensus.cc:513] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.198719 26992 raft_consensus.cc:2391] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5dbf236134d43158377e0310cee23d7 in current term 1: Already voted for candidate ab704ad8154f4a4590e214cf9c1d7c4b in this term.
I20250905 19:57:49.199862 27204 leader_election.cc:290] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 1 election: Requested vote from peers a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:57:49.201587 27141 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "685a60cc8a6841c6808fcba893636187" candidate_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
W20250905 19:57:49.202829 26958 leader_election.cc:343] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:57:49.202615 27068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "685a60cc8a6841c6808fcba893636187" candidate_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5dbf236134d43158377e0310cee23d7"
I20250905 19:57:49.204027 27068 raft_consensus.cc:3058] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:57:49.209896 27068 raft_consensus.cc:2466] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab704ad8154f4a4590e214cf9c1d7c4b in term 1.
W20250905 19:57:49.198168 27033 leader_election.cc:343] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:57:49.210897 27033 leader_election.cc:304] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5dbf236134d43158377e0310cee23d7; no voters: ab704ad8154f4a4590e214cf9c1d7c4b, ac86c30e5d044b9caf64b0c44f5cf7c8
I20250905 19:57:49.211460 26957 leader_election.cc:304] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5dbf236134d43158377e0310cee23d7, ab704ad8154f4a4590e214cf9c1d7c4b; no voters: ac86c30e5d044b9caf64b0c44f5cf7c8
I20250905 19:57:49.211763 27192 raft_consensus.cc:2747] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:57:49.213163 27188 tablet_bootstrap.cc:654] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8: Neither blocks nor log segments found. Creating new log.
I20250905 19:57:49.213317 27033 leader_election.cc:304] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5dbf236134d43158377e0310cee23d7; no voters: ab704ad8154f4a4590e214cf9c1d7c4b, ac86c30e5d044b9caf64b0c44f5cf7c8
I20250905 19:57:49.214226 27192 raft_consensus.cc:2747] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:57:49.214540 27185 tablet_bootstrap.cc:492] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b: No bootstrap required, opened a new log
I20250905 19:57:49.214936 27204 raft_consensus.cc:2802] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:57:49.215134 27185 ts_tablet_manager.cc:1397] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b: Time spent bootstrapping tablet: real 0.106s user 0.013s sys 0.019s
I20250905 19:57:49.216310 27204 raft_consensus.cc:695] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 LEADER]: Becoming Leader. State: Replica: ab704ad8154f4a4590e214cf9c1d7c4b, State: Running, Role: LEADER
I20250905 19:57:49.217828 27204 consensus_queue.cc:237] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.222910 27186 raft_consensus.cc:357] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.223718 27186 raft_consensus.cc:383] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:57:49.224038 27186 raft_consensus.cc:738] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5dbf236134d43158377e0310cee23d7, State: Initialized, Role: FOLLOWER
I20250905 19:57:49.224821 27185 raft_consensus.cc:357] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.225871 27185 raft_consensus.cc:383] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:57:49.226467 27185 raft_consensus.cc:738] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab704ad8154f4a4590e214cf9c1d7c4b, State: Initialized, Role: FOLLOWER
I20250905 19:57:49.224761 27186 consensus_queue.cc:260] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.228768 27188 tablet_bootstrap.cc:492] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8: No bootstrap required, opened a new log
I20250905 19:57:49.229279 27188 ts_tablet_manager.cc:1397] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8: Time spent bootstrapping tablet: real 0.081s user 0.011s sys 0.011s
I20250905 19:57:49.232260 27188 raft_consensus.cc:357] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.233088 27188 raft_consensus.cc:383] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:57:49.233392 27188 raft_consensus.cc:738] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac86c30e5d044b9caf64b0c44f5cf7c8, State: Initialized, Role: FOLLOWER
I20250905 19:57:49.234158 27188 consensus_queue.cc:260] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.235957 27192 raft_consensus.cc:491] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:57:49.236541 27192 raft_consensus.cc:513] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.239857 27188 ts_tablet_manager.cc:1428] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8: Time spent starting tablet: real 0.010s user 0.004s sys 0.004s
I20250905 19:57:49.241057 27188 tablet_bootstrap.cc:492] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8: Bootstrap starting.
I20250905 19:57:49.243736 27186 ts_tablet_manager.cc:1428] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7: Time spent starting tablet: real 0.050s user 0.005s sys 0.004s
I20250905 19:57:49.247975 27188 tablet_bootstrap.cc:654] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8: Neither blocks nor log segments found. Creating new log.
I20250905 19:57:49.248028 27141 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71b161547e6a40e7b93b7f852b76e7d0" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" is_pre_election: true
I20250905 19:57:49.248971 27141 raft_consensus.cc:2466] T 71b161547e6a40e7b93b7f852b76e7d0 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5dbf236134d43158377e0310cee23d7 in term 0.
I20250905 19:57:49.251080 27033 leader_election.cc:304] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5dbf236134d43158377e0310cee23d7, ac86c30e5d044b9caf64b0c44f5cf7c8; no voters:
I20250905 19:57:49.251104 26992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71b161547e6a40e7b93b7f852b76e7d0" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" is_pre_election: true
I20250905 19:57:49.238759 27192 leader_election.cc:290] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:57:49.256712 27185 consensus_queue.cc:260] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.244999 26887 catalog_manager.cc:5582] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b reported cstate change: term changed from 0 to 1, leader changed from <none> to ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129). New cstate: current_term: 1 leader_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } health_report { overall_health: UNKNOWN } } }
I20250905 19:57:49.260628 27188 tablet_bootstrap.cc:492] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8: No bootstrap required, opened a new log
I20250905 19:57:49.257299 27192 raft_consensus.cc:2802] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:57:49.262862 27185 ts_tablet_manager.cc:1428] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b: Time spent starting tablet: real 0.047s user 0.008s sys 0.001s
I20250905 19:57:49.263586 27192 raft_consensus.cc:491] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:57:49.263928 27185 tablet_bootstrap.cc:492] T b2fabc1ff8894320aa02e55c92ecfb1e P ab704ad8154f4a4590e214cf9c1d7c4b: Bootstrap starting.
I20250905 19:57:49.264021 27192 raft_consensus.cc:3058] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:57:49.256070 27186 tablet_bootstrap.cc:492] T c6a00cc443b94edcb46fbfdf1d7ac32c P a5dbf236134d43158377e0310cee23d7: Bootstrap starting.
I20250905 19:57:49.261193 27188 ts_tablet_manager.cc:1397] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.003s
W20250905 19:57:49.267445 27033 leader_election.cc:343] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:57:49.278266 27188 raft_consensus.cc:357] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.282366 27188 raft_consensus.cc:383] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:57:49.282825 27188 raft_consensus.cc:738] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac86c30e5d044b9caf64b0c44f5cf7c8, State: Initialized, Role: FOLLOWER
I20250905 19:57:49.281854 27192 raft_consensus.cc:513] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.285398 27188 consensus_queue.cc:260] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.287518 27185 tablet_bootstrap.cc:654] T b2fabc1ff8894320aa02e55c92ecfb1e P ab704ad8154f4a4590e214cf9c1d7c4b: Neither blocks nor log segments found. Creating new log.
I20250905 19:57:49.288856 27188 ts_tablet_manager.cc:1428] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8: Time spent starting tablet: real 0.019s user 0.010s sys 0.001s
I20250905 19:57:49.289191 26992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71b161547e6a40e7b93b7f852b76e7d0" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
I20250905 19:57:49.290155 27188 tablet_bootstrap.cc:492] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8: Bootstrap starting.
W20250905 19:57:49.290658 27033 leader_election.cc:343] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:57:49.292097 27186 tablet_bootstrap.cc:654] T c6a00cc443b94edcb46fbfdf1d7ac32c P a5dbf236134d43158377e0310cee23d7: Neither blocks nor log segments found. Creating new log.
I20250905 19:57:49.294713 27141 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71b161547e6a40e7b93b7f852b76e7d0" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
I20250905 19:57:49.295444 27141 raft_consensus.cc:3058] T 71b161547e6a40e7b93b7f852b76e7d0 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:57:49.300133 27192 leader_election.cc:290] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 1 election: Requested vote from peers ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:57:49.309121 27141 raft_consensus.cc:2466] T 71b161547e6a40e7b93b7f852b76e7d0 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5dbf236134d43158377e0310cee23d7 in term 1.
I20250905 19:57:49.309743 27188 tablet_bootstrap.cc:654] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8: Neither blocks nor log segments found. Creating new log.
I20250905 19:57:49.312456 27033 leader_election.cc:304] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5dbf236134d43158377e0310cee23d7, ac86c30e5d044b9caf64b0c44f5cf7c8; no voters: ab704ad8154f4a4590e214cf9c1d7c4b
I20250905 19:57:49.313617 27185 tablet_bootstrap.cc:492] T b2fabc1ff8894320aa02e55c92ecfb1e P ab704ad8154f4a4590e214cf9c1d7c4b: No bootstrap required, opened a new log
I20250905 19:57:49.314003 27192 raft_consensus.cc:2802] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:57:49.314199 27185 ts_tablet_manager.cc:1397] T b2fabc1ff8894320aa02e55c92ecfb1e P ab704ad8154f4a4590e214cf9c1d7c4b: Time spent bootstrapping tablet: real 0.050s user 0.018s sys 0.008s
I20250905 19:57:49.314718 27192 raft_consensus.cc:695] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [term 1 LEADER]: Becoming Leader. State: Replica: a5dbf236134d43158377e0310cee23d7, State: Running, Role: LEADER
I20250905 19:57:49.315683 27192 consensus_queue.cc:237] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.318774 27185 raft_consensus.cc:357] T b2fabc1ff8894320aa02e55c92ecfb1e P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.319875 27185 raft_consensus.cc:383] T b2fabc1ff8894320aa02e55c92ecfb1e P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:57:49.320264 27185 raft_consensus.cc:738] T b2fabc1ff8894320aa02e55c92ecfb1e P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab704ad8154f4a4590e214cf9c1d7c4b, State: Initialized, Role: FOLLOWER
I20250905 19:57:49.321147 27185 consensus_queue.cc:260] T b2fabc1ff8894320aa02e55c92ecfb1e P ab704ad8154f4a4590e214cf9c1d7c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.326495 27185 ts_tablet_manager.cc:1428] T b2fabc1ff8894320aa02e55c92ecfb1e P ab704ad8154f4a4590e214cf9c1d7c4b: Time spent starting tablet: real 0.012s user 0.009s sys 0.000s
I20250905 19:57:49.327709 27185 tablet_bootstrap.cc:492] T f6cfad67eeab45ddb61ef84d09f09b6b P ab704ad8154f4a4590e214cf9c1d7c4b: Bootstrap starting.
I20250905 19:57:49.340682 27185 tablet_bootstrap.cc:654] T f6cfad67eeab45ddb61ef84d09f09b6b P ab704ad8154f4a4590e214cf9c1d7c4b: Neither blocks nor log segments found. Creating new log.
I20250905 19:57:49.340166 26887 catalog_manager.cc:5582] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5dbf236134d43158377e0310cee23d7 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "a5dbf236134d43158377e0310cee23d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } health_report { overall_health: UNKNOWN } } }
I20250905 19:57:49.345305 27188 tablet_bootstrap.cc:492] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8: No bootstrap required, opened a new log
I20250905 19:57:49.345851 27188 ts_tablet_manager.cc:1397] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8: Time spent bootstrapping tablet: real 0.056s user 0.009s sys 0.008s
I20250905 19:57:49.350951 27188 raft_consensus.cc:357] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.351706 27188 raft_consensus.cc:383] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:57:49.352039 27186 tablet_bootstrap.cc:492] T c6a00cc443b94edcb46fbfdf1d7ac32c P a5dbf236134d43158377e0310cee23d7: No bootstrap required, opened a new log
I20250905 19:57:49.352005 27188 raft_consensus.cc:738] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac86c30e5d044b9caf64b0c44f5cf7c8, State: Initialized, Role: FOLLOWER
I20250905 19:57:49.352581 27186 ts_tablet_manager.cc:1397] T c6a00cc443b94edcb46fbfdf1d7ac32c P a5dbf236134d43158377e0310cee23d7: Time spent bootstrapping tablet: real 0.097s user 0.012s sys 0.006s
I20250905 19:57:49.352834 27188 consensus_queue.cc:260] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.355742 27188 ts_tablet_manager.cc:1428] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8: Time spent starting tablet: real 0.010s user 0.006s sys 0.002s
I20250905 19:57:49.356895 27188 tablet_bootstrap.cc:492] T 685a60cc8a6841c6808fcba893636187 P ac86c30e5d044b9caf64b0c44f5cf7c8: Bootstrap starting.
I20250905 19:57:49.364423 27188 tablet_bootstrap.cc:654] T 685a60cc8a6841c6808fcba893636187 P ac86c30e5d044b9caf64b0c44f5cf7c8: Neither blocks nor log segments found. Creating new log.
I20250905 19:57:49.370131 27186 raft_consensus.cc:357] T c6a00cc443b94edcb46fbfdf1d7ac32c P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.371064 27186 raft_consensus.cc:383] T c6a00cc443b94edcb46fbfdf1d7ac32c P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:57:49.371536 27186 raft_consensus.cc:738] T c6a00cc443b94edcb46fbfdf1d7ac32c P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5dbf236134d43158377e0310cee23d7, State: Initialized, Role: FOLLOWER
I20250905 19:57:49.372457 27186 consensus_queue.cc:260] T c6a00cc443b94edcb46fbfdf1d7ac32c P a5dbf236134d43158377e0310cee23d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.375115 27186 ts_tablet_manager.cc:1428] T c6a00cc443b94edcb46fbfdf1d7ac32c P a5dbf236134d43158377e0310cee23d7: Time spent starting tablet: real 0.022s user 0.005s sys 0.004s
I20250905 19:57:49.375937 27186 tablet_bootstrap.cc:492] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7: Bootstrap starting.
I20250905 19:57:49.386183 27186 tablet_bootstrap.cc:654] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7: Neither blocks nor log segments found. Creating new log.
I20250905 19:57:49.381641 27185 tablet_bootstrap.cc:492] T f6cfad67eeab45ddb61ef84d09f09b6b P ab704ad8154f4a4590e214cf9c1d7c4b: No bootstrap required, opened a new log
I20250905 19:57:49.391906 27185 ts_tablet_manager.cc:1397] T f6cfad67eeab45ddb61ef84d09f09b6b P ab704ad8154f4a4590e214cf9c1d7c4b: Time spent bootstrapping tablet: real 0.064s user 0.011s sys 0.025s
I20250905 19:57:49.394825 27188 tablet_bootstrap.cc:492] T 685a60cc8a6841c6808fcba893636187 P ac86c30e5d044b9caf64b0c44f5cf7c8: No bootstrap required, opened a new log
I20250905 19:57:49.395467 27188 ts_tablet_manager.cc:1397] T 685a60cc8a6841c6808fcba893636187 P ac86c30e5d044b9caf64b0c44f5cf7c8: Time spent bootstrapping tablet: real 0.039s user 0.020s sys 0.004s
I20250905 19:57:49.395285 27185 raft_consensus.cc:357] T f6cfad67eeab45ddb61ef84d09f09b6b P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.396306 27185 raft_consensus.cc:383] T f6cfad67eeab45ddb61ef84d09f09b6b P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:57:49.396723 27185 raft_consensus.cc:738] T f6cfad67eeab45ddb61ef84d09f09b6b P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab704ad8154f4a4590e214cf9c1d7c4b, State: Initialized, Role: FOLLOWER
I20250905 19:57:49.397806 27186 tablet_bootstrap.cc:492] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7: No bootstrap required, opened a new log
I20250905 19:57:49.397646 27185 consensus_queue.cc:260] T f6cfad67eeab45ddb61ef84d09f09b6b P ab704ad8154f4a4590e214cf9c1d7c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.398351 27186 ts_tablet_manager.cc:1397] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.008s
I20250905 19:57:49.398808 27188 raft_consensus.cc:357] T 685a60cc8a6841c6808fcba893636187 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.401137 27185 ts_tablet_manager.cc:1428] T f6cfad67eeab45ddb61ef84d09f09b6b P ab704ad8154f4a4590e214cf9c1d7c4b: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20250905 19:57:49.401309 27188 raft_consensus.cc:383] T 685a60cc8a6841c6808fcba893636187 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:57:49.401705 27188 raft_consensus.cc:738] T 685a60cc8a6841c6808fcba893636187 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac86c30e5d044b9caf64b0c44f5cf7c8, State: Initialized, Role: FOLLOWER
I20250905 19:57:49.402593 27185 tablet_bootstrap.cc:492] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b: Bootstrap starting.
I20250905 19:57:49.402485 27188 consensus_queue.cc:260] T 685a60cc8a6841c6808fcba893636187 P ac86c30e5d044b9caf64b0c44f5cf7c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.405393 27188 ts_tablet_manager.cc:1428] T 685a60cc8a6841c6808fcba893636187 P ac86c30e5d044b9caf64b0c44f5cf7c8: Time spent starting tablet: real 0.010s user 0.001s sys 0.007s
I20250905 19:57:49.406471 27188 tablet_bootstrap.cc:492] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8: Bootstrap starting.
I20250905 19:57:49.414258 27185 tablet_bootstrap.cc:654] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b: Neither blocks nor log segments found. Creating new log.
I20250905 19:57:49.414634 27188 tablet_bootstrap.cc:654] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8: Neither blocks nor log segments found. Creating new log.
I20250905 19:57:49.429450 27186 raft_consensus.cc:357] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.430327 27186 raft_consensus.cc:383] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:57:49.430675 27186 raft_consensus.cc:738] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5dbf236134d43158377e0310cee23d7, State: Initialized, Role: FOLLOWER
I20250905 19:57:49.431512 27186 consensus_queue.cc:260] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.432387 27192 raft_consensus.cc:491] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:57:49.432775 27192 raft_consensus.cc:513] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.433991 27186 ts_tablet_manager.cc:1428] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250905 19:57:49.435463 27186 tablet_bootstrap.cc:492] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7: Bootstrap starting.
I20250905 19:57:49.437940 26992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2fabc1ff8894320aa02e55c92ecfb1e" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" is_pre_election: true
I20250905 19:57:49.439136 26992 raft_consensus.cc:2466] T b2fabc1ff8894320aa02e55c92ecfb1e P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5dbf236134d43158377e0310cee23d7 in term 0.
I20250905 19:57:49.440505 27033 leader_election.cc:304] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5dbf236134d43158377e0310cee23d7, ab704ad8154f4a4590e214cf9c1d7c4b; no voters:
I20250905 19:57:49.441632 27216 raft_consensus.cc:2802] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:57:49.442293 27216 raft_consensus.cc:491] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:57:49.442713 27216 raft_consensus.cc:3058] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:57:49.443306 27186 tablet_bootstrap.cc:654] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7: Neither blocks nor log segments found. Creating new log.
I20250905 19:57:49.447082 27192 leader_election.cc:290] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:57:49.448310 27193 raft_consensus.cc:491] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:57:49.449143 27193 raft_consensus.cc:513] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.451881 27185 tablet_bootstrap.cc:492] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b: No bootstrap required, opened a new log
I20250905 19:57:49.452210 27141 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2fabc1ff8894320aa02e55c92ecfb1e" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" is_pre_election: true
I20250905 19:57:49.452524 27185 ts_tablet_manager.cc:1397] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.018s
I20250905 19:57:49.452931 27141 raft_consensus.cc:2466] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5dbf236134d43158377e0310cee23d7 in term 0.
I20250905 19:57:49.456043 27185 raft_consensus.cc:357] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.457067 27185 raft_consensus.cc:383] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:57:49.457468 27185 raft_consensus.cc:738] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab704ad8154f4a4590e214cf9c1d7c4b, State: Initialized, Role: FOLLOWER
I20250905 19:57:49.458459 27185 consensus_queue.cc:260] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.460032 26992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6a00cc443b94edcb46fbfdf1d7ac32c" candidate_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" is_pre_election: true
I20250905 19:57:49.460776 26992 raft_consensus.cc:2466] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac86c30e5d044b9caf64b0c44f5cf7c8 in term 0.
I20250905 19:57:49.461692 27185 ts_tablet_manager.cc:1428] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250905 19:57:49.462050 27107 leader_election.cc:304] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab704ad8154f4a4590e214cf9c1d7c4b, ac86c30e5d044b9caf64b0c44f5cf7c8; no voters:
I20250905 19:57:49.463073 27185 tablet_bootstrap.cc:492] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b: Bootstrap starting.
I20250905 19:57:49.463934 27068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6a00cc443b94edcb46fbfdf1d7ac32c" candidate_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5dbf236134d43158377e0310cee23d7" is_pre_election: true
I20250905 19:57:49.464638 27068 raft_consensus.cc:2466] T c6a00cc443b94edcb46fbfdf1d7ac32c P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac86c30e5d044b9caf64b0c44f5cf7c8 in term 0.
I20250905 19:57:49.466714 27193 leader_election.cc:290] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107), a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875)
I20250905 19:57:49.467453 27193 raft_consensus.cc:2802] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:57:49.467900 27193 raft_consensus.cc:491] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:57:49.470582 27186 tablet_bootstrap.cc:492] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7: No bootstrap required, opened a new log
I20250905 19:57:49.471194 27186 ts_tablet_manager.cc:1397] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7: Time spent bootstrapping tablet: real 0.036s user 0.015s sys 0.008s
I20250905 19:57:49.472340 27185 tablet_bootstrap.cc:654] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b: Neither blocks nor log segments found. Creating new log.
I20250905 19:57:49.468412 27193 raft_consensus.cc:3058] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:57:49.479180 27188 tablet_bootstrap.cc:492] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8: No bootstrap required, opened a new log
I20250905 19:57:49.479650 27188 ts_tablet_manager.cc:1397] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8: Time spent bootstrapping tablet: real 0.073s user 0.008s sys 0.011s
I20250905 19:57:49.481613 27186 raft_consensus.cc:357] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.482416 27186 raft_consensus.cc:383] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:57:49.482765 27186 raft_consensus.cc:738] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5dbf236134d43158377e0310cee23d7, State: Initialized, Role: FOLLOWER
I20250905 19:57:49.483311 27188 raft_consensus.cc:357] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.484086 27188 raft_consensus.cc:383] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:57:49.484601 27188 raft_consensus.cc:738] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac86c30e5d044b9caf64b0c44f5cf7c8, State: Initialized, Role: FOLLOWER
I20250905 19:57:49.484355 27186 consensus_queue.cc:260] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.485824 27188 consensus_queue.cc:260] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.483866 27216 raft_consensus.cc:513] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.491930 27188 ts_tablet_manager.cc:1428] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20250905 19:57:49.493041 27188 tablet_bootstrap.cc:492] T 7640f63d346f4d078ecb8d0e8a6c5670 P ac86c30e5d044b9caf64b0c44f5cf7c8: Bootstrap starting.
I20250905 19:57:49.495240 26992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2fabc1ff8894320aa02e55c92ecfb1e" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
I20250905 19:57:49.496008 27185 tablet_bootstrap.cc:492] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b: No bootstrap required, opened a new log
I20250905 19:57:49.496639 27185 ts_tablet_manager.cc:1397] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b: Time spent bootstrapping tablet: real 0.034s user 0.015s sys 0.000s
I20250905 19:57:49.496686 27141 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2fabc1ff8894320aa02e55c92ecfb1e" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
I20250905 19:57:49.495947 26992 raft_consensus.cc:3058] T b2fabc1ff8894320aa02e55c92ecfb1e P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:57:49.492583 27216 leader_election.cc:290] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 1 election: Requested vote from peers ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:57:49.502267 27141 raft_consensus.cc:3058] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:57:49.503628 27186 ts_tablet_manager.cc:1428] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7: Time spent starting tablet: real 0.032s user 0.007s sys 0.000s
I20250905 19:57:49.506053 26992 raft_consensus.cc:2466] T b2fabc1ff8894320aa02e55c92ecfb1e P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5dbf236134d43158377e0310cee23d7 in term 1.
I20250905 19:57:49.506654 27186 tablet_bootstrap.cc:492] T f6cfad67eeab45ddb61ef84d09f09b6b P a5dbf236134d43158377e0310cee23d7: Bootstrap starting.
I20250905 19:57:49.507431 27033 leader_election.cc:304] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5dbf236134d43158377e0310cee23d7, ab704ad8154f4a4590e214cf9c1d7c4b; no voters:
I20250905 19:57:49.504719 27193 raft_consensus.cc:513] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.508503 27216 raft_consensus.cc:2802] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:57:49.509231 27216 raft_consensus.cc:695] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 [term 1 LEADER]: Becoming Leader. State: Replica: a5dbf236134d43158377e0310cee23d7, State: Running, Role: LEADER
I20250905 19:57:49.510079 27216 consensus_queue.cc:237] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.512629 27068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6a00cc443b94edcb46fbfdf1d7ac32c" candidate_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5dbf236134d43158377e0310cee23d7"
I20250905 19:57:49.518668 27193 leader_election.cc:290] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 1 election: Requested vote from peers ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107), a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875)
I20250905 19:57:49.518791 27230 raft_consensus.cc:491] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:57:49.522020 27193 raft_consensus.cc:491] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:57:49.518673 27068 raft_consensus.cc:3058] T c6a00cc443b94edcb46fbfdf1d7ac32c P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:57:49.525233 27193 raft_consensus.cc:513] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.525121 27230 raft_consensus.cc:513] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.528903 27193 leader_election.cc:290] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107), a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875)
I20250905 19:57:49.532397 27067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6cfad67eeab45ddb61ef84d09f09b6b" candidate_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5dbf236134d43158377e0310cee23d7" is_pre_election: true
W20250905 19:57:49.534587 27106 leader_election.cc:343] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:57:49.536136 27230 leader_election.cc:290] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107), a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875)
I20250905 19:57:49.540773 27068 raft_consensus.cc:2466] T c6a00cc443b94edcb46fbfdf1d7ac32c P a5dbf236134d43158377e0310cee23d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac86c30e5d044b9caf64b0c44f5cf7c8 in term 1.
W20250905 19:57:49.541301 26938 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250905 19:57:49.542464 27106 leader_election.cc:304] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5dbf236134d43158377e0310cee23d7, ac86c30e5d044b9caf64b0c44f5cf7c8; no voters:
I20250905 19:57:49.544039 27230 raft_consensus.cc:2802] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:57:49.545004 27230 raft_consensus.cc:695] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 LEADER]: Becoming Leader. State: Replica: ac86c30e5d044b9caf64b0c44f5cf7c8, State: Running, Role: LEADER
I20250905 19:57:49.546058 27230 consensus_queue.cc:237] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.548614 26991 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbaec702dd3f48f2a1fcf12f8befed7e" candidate_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" is_pre_election: true
I20250905 19:57:49.555869 26993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6a00cc443b94edcb46fbfdf1d7ac32c" candidate_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
W20250905 19:57:49.556478 27107 leader_election.cc:343] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:57:49.556661 26993 raft_consensus.cc:3058] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:57:49.557699 27141 raft_consensus.cc:2466] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5dbf236134d43158377e0310cee23d7 in term 1.
I20250905 19:57:49.560492 27188 tablet_bootstrap.cc:654] T 7640f63d346f4d078ecb8d0e8a6c5670 P ac86c30e5d044b9caf64b0c44f5cf7c8: Neither blocks nor log segments found. Creating new log.
I20250905 19:57:49.559008 27067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbaec702dd3f48f2a1fcf12f8befed7e" candidate_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5dbf236134d43158377e0310cee23d7" is_pre_election: true
I20250905 19:57:49.561136 27067 raft_consensus.cc:2466] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac86c30e5d044b9caf64b0c44f5cf7c8 in term 0.
I20250905 19:57:49.563674 27106 leader_election.cc:304] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5dbf236134d43158377e0310cee23d7, ac86c30e5d044b9caf64b0c44f5cf7c8; no voters: ab704ad8154f4a4590e214cf9c1d7c4b
I20250905 19:57:49.559325 26992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6cfad67eeab45ddb61ef84d09f09b6b" candidate_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" is_pre_election: true
I20250905 19:57:49.565299 27232 raft_consensus.cc:2802] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:57:49.565516 26992 raft_consensus.cc:2466] T f6cfad67eeab45ddb61ef84d09f09b6b P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac86c30e5d044b9caf64b0c44f5cf7c8 in term 0.
I20250905 19:57:49.565763 27232 raft_consensus.cc:491] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:57:49.567131 27107 leader_election.cc:304] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab704ad8154f4a4590e214cf9c1d7c4b, ac86c30e5d044b9caf64b0c44f5cf7c8; no voters: a5dbf236134d43158377e0310cee23d7
I20250905 19:57:49.567972 27193 raft_consensus.cc:2802] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:57:49.568338 27193 raft_consensus.cc:491] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:57:49.569084 27193 raft_consensus.cc:3058] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:57:49.566416 27232 raft_consensus.cc:3058] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:57:49.576951 26993 raft_consensus.cc:2466] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac86c30e5d044b9caf64b0c44f5cf7c8 in term 1.
I20250905 19:57:49.580659 27188 tablet_bootstrap.cc:492] T 7640f63d346f4d078ecb8d0e8a6c5670 P ac86c30e5d044b9caf64b0c44f5cf7c8: No bootstrap required, opened a new log
I20250905 19:57:49.581965 27188 ts_tablet_manager.cc:1397] T 7640f63d346f4d078ecb8d0e8a6c5670 P ac86c30e5d044b9caf64b0c44f5cf7c8: Time spent bootstrapping tablet: real 0.089s user 0.015s sys 0.000s
I20250905 19:57:49.590979 27204 raft_consensus.cc:491] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:57:49.591531 27204 raft_consensus.cc:513] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.593590 27204 leader_election.cc:290] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:57:49.594055 26887 catalog_manager.cc:5582] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5dbf236134d43158377e0310cee23d7 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "a5dbf236134d43158377e0310cee23d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } health_report { overall_health: UNKNOWN } } }
I20250905 19:57:49.594942 27067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c08349e8a8394ee182d5d0928074181f" candidate_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5dbf236134d43158377e0310cee23d7" is_pre_election: true
W20250905 19:57:49.597012 26957 leader_election.cc:343] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:57:49.597168 27232 raft_consensus.cc:513] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.598356 27141 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c08349e8a8394ee182d5d0928074181f" candidate_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" is_pre_election: true
I20250905 19:57:49.598570 27185 raft_consensus.cc:357] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.599601 27185 raft_consensus.cc:383] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:57:49.599391 27141 raft_consensus.cc:2466] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab704ad8154f4a4590e214cf9c1d7c4b in term 0.
I20250905 19:57:49.599944 27185 raft_consensus.cc:738] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab704ad8154f4a4590e214cf9c1d7c4b, State: Initialized, Role: FOLLOWER
I20250905 19:57:49.592492 27193 raft_consensus.cc:513] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.604194 27193 leader_election.cc:290] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 1 election: Requested vote from peers ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107), a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875)
I20250905 19:57:49.607242 27068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbaec702dd3f48f2a1fcf12f8befed7e" candidate_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5dbf236134d43158377e0310cee23d7"
I20250905 19:57:49.603996 27185 consensus_queue.cc:260] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.605445 27067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6cfad67eeab45ddb61ef84d09f09b6b" candidate_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5dbf236134d43158377e0310cee23d7"
I20250905 19:57:49.607873 27068 raft_consensus.cc:3058] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:57:49.608915 27106 leader_election.cc:343] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:57:49.605554 26958 leader_election.cc:304] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab704ad8154f4a4590e214cf9c1d7c4b, ac86c30e5d044b9caf64b0c44f5cf7c8; no voters: a5dbf236134d43158377e0310cee23d7
I20250905 19:57:49.606107 27232 leader_election.cc:290] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 1 election: Requested vote from peers ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107), a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875)
I20250905 19:57:49.606590 27186 tablet_bootstrap.cc:654] T f6cfad67eeab45ddb61ef84d09f09b6b P a5dbf236134d43158377e0310cee23d7: Neither blocks nor log segments found. Creating new log.
I20250905 19:57:49.606940 27188 raft_consensus.cc:357] T 7640f63d346f4d078ecb8d0e8a6c5670 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.612658 27188 raft_consensus.cc:383] T 7640f63d346f4d078ecb8d0e8a6c5670 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:57:49.614583 27204 raft_consensus.cc:2802] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:57:49.615149 27204 raft_consensus.cc:491] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:57:49.615648 27204 raft_consensus.cc:3058] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:57:49.613292 27188 raft_consensus.cc:738] T 7640f63d346f4d078ecb8d0e8a6c5670 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac86c30e5d044b9caf64b0c44f5cf7c8, State: Initialized, Role: FOLLOWER
I20250905 19:57:49.607566 27193 raft_consensus.cc:491] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:57:49.618269 27193 raft_consensus.cc:513] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.621208 27193 leader_election.cc:290] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107), a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875)
I20250905 19:57:49.616175 27185 ts_tablet_manager.cc:1428] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b: Time spent starting tablet: real 0.119s user 0.015s sys 0.004s
I20250905 19:57:49.623875 27067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f9344374d5649489c2f4336e9a1b15b" candidate_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5dbf236134d43158377e0310cee23d7" is_pre_election: true
I20250905 19:57:49.624444 27188 consensus_queue.cc:260] T 7640f63d346f4d078ecb8d0e8a6c5670 P ac86c30e5d044b9caf64b0c44f5cf7c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.627907 27067 raft_consensus.cc:2466] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac86c30e5d044b9caf64b0c44f5cf7c8 in term 0.
I20250905 19:57:49.630373 27185 tablet_bootstrap.cc:492] T dbaec702dd3f48f2a1fcf12f8befed7e P ab704ad8154f4a4590e214cf9c1d7c4b: Bootstrap starting.
I20250905 19:57:49.630620 27106 leader_election.cc:304] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5dbf236134d43158377e0310cee23d7, ac86c30e5d044b9caf64b0c44f5cf7c8; no voters:
I20250905 19:57:49.631544 27141 raft_consensus.cc:3058] T 685a60cc8a6841c6808fcba893636187 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:57:49.636324 27068 raft_consensus.cc:2466] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac86c30e5d044b9caf64b0c44f5cf7c8 in term 1.
I20250905 19:57:49.637099 27185 tablet_bootstrap.cc:654] T dbaec702dd3f48f2a1fcf12f8befed7e P ab704ad8154f4a4590e214cf9c1d7c4b: Neither blocks nor log segments found. Creating new log.
I20250905 19:57:49.639168 27204 raft_consensus.cc:513] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.646328 27106 leader_election.cc:304] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5dbf236134d43158377e0310cee23d7, ac86c30e5d044b9caf64b0c44f5cf7c8; no voters:
I20250905 19:57:49.647317 27232 raft_consensus.cc:2802] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:57:49.647800 27232 raft_consensus.cc:695] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 LEADER]: Becoming Leader. State: Replica: ac86c30e5d044b9caf64b0c44f5cf7c8, State: Running, Role: LEADER
I20250905 19:57:49.648075 26993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6cfad67eeab45ddb61ef84d09f09b6b" candidate_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
I20250905 19:57:49.648330 26992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbaec702dd3f48f2a1fcf12f8befed7e" candidate_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
I20250905 19:57:49.648805 26993 raft_consensus.cc:3058] T f6cfad67eeab45ddb61ef84d09f09b6b P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:57:49.648634 27232 consensus_queue.cc:237] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.649514 26991 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f9344374d5649489c2f4336e9a1b15b" candidate_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" is_pre_election: true
I20250905 19:57:49.650127 26991 raft_consensus.cc:2466] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac86c30e5d044b9caf64b0c44f5cf7c8 in term 0.
W20250905 19:57:49.653112 27107 leader_election.cc:343] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:57:49.654072 27204 leader_election.cc:290] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 1 election: Requested vote from peers a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:57:49.656594 27142 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c08349e8a8394ee182d5d0928074181f" candidate_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
I20250905 19:57:49.657263 27142 raft_consensus.cc:3058] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:57:49.643002 27193 raft_consensus.cc:2802] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:57:49.659559 27204 consensus_queue.cc:1035] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:57:49.658016 27193 raft_consensus.cc:491] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:57:49.662477 27193 raft_consensus.cc:3058] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:57:49.665550 27142 raft_consensus.cc:2466] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab704ad8154f4a4590e214cf9c1d7c4b in term 1.
I20250905 19:57:49.666934 26958 leader_election.cc:304] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab704ad8154f4a4590e214cf9c1d7c4b, ac86c30e5d044b9caf64b0c44f5cf7c8; no voters:
I20250905 19:57:49.667814 27204 raft_consensus.cc:2802] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:57:49.669059 27204 raft_consensus.cc:695] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 LEADER]: Becoming Leader. State: Replica: ab704ad8154f4a4590e214cf9c1d7c4b, State: Running, Role: LEADER
I20250905 19:57:49.673420 27188 ts_tablet_manager.cc:1428] T 7640f63d346f4d078ecb8d0e8a6c5670 P ac86c30e5d044b9caf64b0c44f5cf7c8: Time spent starting tablet: real 0.091s user 0.009s sys 0.000s
I20250905 19:57:49.672547 27068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c08349e8a8394ee182d5d0928074181f" candidate_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5dbf236134d43158377e0310cee23d7"
I20250905 19:57:49.655759 26888 catalog_manager.cc:5582] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131). New cstate: current_term: 1 leader_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } health_report { overall_health: HEALTHY } } }
I20250905 19:57:49.670296 27204 consensus_queue.cc:237] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.683472 26993 raft_consensus.cc:2466] T f6cfad67eeab45ddb61ef84d09f09b6b P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac86c30e5d044b9caf64b0c44f5cf7c8 in term 1.
W20250905 19:57:49.686883 26957 leader_election.cc:343] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:57:49.689258 27185 tablet_bootstrap.cc:492] T dbaec702dd3f48f2a1fcf12f8befed7e P ab704ad8154f4a4590e214cf9c1d7c4b: No bootstrap required, opened a new log
I20250905 19:57:49.689729 27185 ts_tablet_manager.cc:1397] T dbaec702dd3f48f2a1fcf12f8befed7e P ab704ad8154f4a4590e214cf9c1d7c4b: Time spent bootstrapping tablet: real 0.060s user 0.015s sys 0.031s
I20250905 19:57:49.688519 27107 leader_election.cc:304] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab704ad8154f4a4590e214cf9c1d7c4b, ac86c30e5d044b9caf64b0c44f5cf7c8; no voters: a5dbf236134d43158377e0310cee23d7
I20250905 19:57:49.690738 27232 raft_consensus.cc:2802] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:57:49.698961 27186 tablet_bootstrap.cc:492] T f6cfad67eeab45ddb61ef84d09f09b6b P a5dbf236134d43158377e0310cee23d7: No bootstrap required, opened a new log
I20250905 19:57:49.699538 27186 ts_tablet_manager.cc:1397] T f6cfad67eeab45ddb61ef84d09f09b6b P a5dbf236134d43158377e0310cee23d7: Time spent bootstrapping tablet: real 0.193s user 0.011s sys 0.019s
I20250905 19:57:49.694772 27216 raft_consensus.cc:491] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:57:49.700423 27216 raft_consensus.cc:513] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.702366 27216 leader_election.cc:290] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:57:49.699541 27232 raft_consensus.cc:695] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 LEADER]: Becoming Leader. State: Replica: ac86c30e5d044b9caf64b0c44f5cf7c8, State: Running, Role: LEADER
I20250905 19:57:49.693557 27185 raft_consensus.cc:357] T dbaec702dd3f48f2a1fcf12f8befed7e P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.704844 27186 raft_consensus.cc:357] T f6cfad67eeab45ddb61ef84d09f09b6b P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.703769 27232 consensus_queue.cc:237] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.706881 27186 raft_consensus.cc:383] T f6cfad67eeab45ddb61ef84d09f09b6b P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:57:49.706346 26890 catalog_manager.cc:5582] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b reported cstate change: term changed from 0 to 1, leader changed from <none> to ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129). New cstate: current_term: 1 leader_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } health_report { overall_health: UNKNOWN } } }
I20250905 19:57:49.707249 27186 raft_consensus.cc:738] T f6cfad67eeab45ddb61ef84d09f09b6b P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5dbf236134d43158377e0310cee23d7, State: Initialized, Role: FOLLOWER
I20250905 19:57:49.705762 27142 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7640f63d346f4d078ecb8d0e8a6c5670" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" is_pre_election: true
I20250905 19:57:49.712263 27142 raft_consensus.cc:2466] T 7640f63d346f4d078ecb8d0e8a6c5670 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5dbf236134d43158377e0310cee23d7 in term 0.
I20250905 19:57:49.705255 27185 raft_consensus.cc:383] T dbaec702dd3f48f2a1fcf12f8befed7e P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:57:49.713551 27185 raft_consensus.cc:738] T dbaec702dd3f48f2a1fcf12f8befed7e P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab704ad8154f4a4590e214cf9c1d7c4b, State: Initialized, Role: FOLLOWER
I20250905 19:57:49.715643 27185 consensus_queue.cc:260] T dbaec702dd3f48f2a1fcf12f8befed7e P ab704ad8154f4a4590e214cf9c1d7c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.715822 27186 consensus_queue.cc:260] T f6cfad67eeab45ddb61ef84d09f09b6b P a5dbf236134d43158377e0310cee23d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.703742 26887 catalog_manager.cc:5582] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131). New cstate: current_term: 1 leader_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } health_report { overall_health: HEALTHY } } }
I20250905 19:57:49.716850 27033 leader_election.cc:304] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5dbf236134d43158377e0310cee23d7, ac86c30e5d044b9caf64b0c44f5cf7c8; no voters:
I20250905 19:57:49.705035 26993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7640f63d346f4d078ecb8d0e8a6c5670" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" is_pre_election: true
I20250905 19:57:49.724977 27192 raft_consensus.cc:2802] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:57:49.725502 27192 raft_consensus.cc:491] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:57:49.725841 27192 raft_consensus.cc:3058] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:57:49.726552 26993 raft_consensus.cc:2466] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5dbf236134d43158377e0310cee23d7 in term 0.
I20250905 19:57:49.705127 27193 raft_consensus.cc:513] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
W20250905 19:57:49.730476 27033 consensus_peers.cc:489] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 -> Peer ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107): Couldn't send request to peer ab704ad8154f4a4590e214cf9c1d7c4b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:57:49.732194 27204 consensus_queue.cc:1035] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:57:49.734380 26993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f9344374d5649489c2f4336e9a1b15b" candidate_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
I20250905 19:57:49.735356 26993 raft_consensus.cc:3058] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:57:49.737013 27192 raft_consensus.cc:513] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.740474 26991 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7640f63d346f4d078ecb8d0e8a6c5670" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
I20250905 19:57:49.741170 26991 raft_consensus.cc:3058] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:57:49.742995 27186 ts_tablet_manager.cc:1428] T f6cfad67eeab45ddb61ef84d09f09b6b P a5dbf236134d43158377e0310cee23d7: Time spent starting tablet: real 0.043s user 0.008s sys 0.000s
I20250905 19:57:49.744117 27186 tablet_bootstrap.cc:492] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7: Bootstrap starting.
I20250905 19:57:49.752941 27216 consensus_queue.cc:1035] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:57:49.760481 27186 tablet_bootstrap.cc:654] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7: Neither blocks nor log segments found. Creating new log.
I20250905 19:57:49.760289 27067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f9344374d5649489c2f4336e9a1b15b" candidate_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5dbf236134d43158377e0310cee23d7"
I20250905 19:57:49.761070 27067 raft_consensus.cc:3058] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:57:49.768172 26991 raft_consensus.cc:2466] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5dbf236134d43158377e0310cee23d7 in term 1.
I20250905 19:57:49.769749 27033 leader_election.cc:304] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5dbf236134d43158377e0310cee23d7, ab704ad8154f4a4590e214cf9c1d7c4b; no voters:
I20250905 19:57:49.770818 27216 raft_consensus.cc:2802] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:57:49.770172 27067 raft_consensus.cc:2466] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac86c30e5d044b9caf64b0c44f5cf7c8 in term 1.
I20250905 19:57:49.774008 27193 leader_election.cc:290] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 1 election: Requested vote from peers ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107), a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875)
I20250905 19:57:49.780057 27106 leader_election.cc:304] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5dbf236134d43158377e0310cee23d7, ac86c30e5d044b9caf64b0c44f5cf7c8; no voters:
I20250905 19:57:49.781592 27193 raft_consensus.cc:2802] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:57:49.783104 27216 raft_consensus.cc:695] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 1 LEADER]: Becoming Leader. State: Replica: a5dbf236134d43158377e0310cee23d7, State: Running, Role: LEADER
I20250905 19:57:49.806142 26993 raft_consensus.cc:2466] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac86c30e5d044b9caf64b0c44f5cf7c8 in term 1.
I20250905 19:57:49.809541 27192 leader_election.cc:290] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 1 election: Requested vote from peers ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:57:49.783823 27193 raft_consensus.cc:695] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 LEADER]: Becoming Leader. State: Replica: ac86c30e5d044b9caf64b0c44f5cf7c8, State: Running, Role: LEADER
I20250905 19:57:49.811087 27193 consensus_queue.cc:237] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.810252 27141 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7640f63d346f4d078ecb8d0e8a6c5670" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
I20250905 19:57:49.814469 27141 raft_consensus.cc:3058] T 7640f63d346f4d078ecb8d0e8a6c5670 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:57:49.814690 27216 consensus_queue.cc:237] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.841866 27186 tablet_bootstrap.cc:492] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7: No bootstrap required, opened a new log
I20250905 19:57:49.842360 27186 ts_tablet_manager.cc:1397] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7: Time spent bootstrapping tablet: real 0.098s user 0.032s sys 0.008s
I20250905 19:57:49.845786 27186 raft_consensus.cc:357] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.846598 27141 raft_consensus.cc:2466] T 7640f63d346f4d078ecb8d0e8a6c5670 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5dbf236134d43158377e0310cee23d7 in term 1.
I20250905 19:57:49.842991 26888 catalog_manager.cc:5582] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131). New cstate: current_term: 1 leader_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } health_report { overall_health: HEALTHY } } }
I20250905 19:57:49.846884 27186 raft_consensus.cc:383] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:57:49.849713 27186 raft_consensus.cc:738] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5dbf236134d43158377e0310cee23d7, State: Initialized, Role: FOLLOWER
I20250905 19:57:49.851012 27186 consensus_queue.cc:260] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.855675 27185 ts_tablet_manager.cc:1428] T dbaec702dd3f48f2a1fcf12f8befed7e P ab704ad8154f4a4590e214cf9c1d7c4b: Time spent starting tablet: real 0.166s user 0.027s sys 0.022s
I20250905 19:57:49.856849 27185 tablet_bootstrap.cc:492] T 71b161547e6a40e7b93b7f852b76e7d0 P ab704ad8154f4a4590e214cf9c1d7c4b: Bootstrap starting.
I20250905 19:57:49.864791 27185 tablet_bootstrap.cc:654] T 71b161547e6a40e7b93b7f852b76e7d0 P ab704ad8154f4a4590e214cf9c1d7c4b: Neither blocks nor log segments found. Creating new log.
I20250905 19:57:49.873304 27186 ts_tablet_manager.cc:1428] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7: Time spent starting tablet: real 0.031s user 0.008s sys 0.000s
I20250905 19:57:49.872486 26889 catalog_manager.cc:5582] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131). New cstate: current_term: 1 leader_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } health_report { overall_health: HEALTHY } } }
I20250905 19:57:49.882470 27185 tablet_bootstrap.cc:492] T 71b161547e6a40e7b93b7f852b76e7d0 P ab704ad8154f4a4590e214cf9c1d7c4b: No bootstrap required, opened a new log
I20250905 19:57:49.882964 27185 ts_tablet_manager.cc:1397] T 71b161547e6a40e7b93b7f852b76e7d0 P ab704ad8154f4a4590e214cf9c1d7c4b: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.010s
I20250905 19:57:49.889093 27185 raft_consensus.cc:357] T 71b161547e6a40e7b93b7f852b76e7d0 P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.889824 27185 raft_consensus.cc:383] T 71b161547e6a40e7b93b7f852b76e7d0 P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:57:49.890228 27185 raft_consensus.cc:738] T 71b161547e6a40e7b93b7f852b76e7d0 P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab704ad8154f4a4590e214cf9c1d7c4b, State: Initialized, Role: FOLLOWER
I20250905 19:57:49.890909 27185 consensus_queue.cc:260] T 71b161547e6a40e7b93b7f852b76e7d0 P ab704ad8154f4a4590e214cf9c1d7c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.893079 27185 ts_tablet_manager.cc:1428] T 71b161547e6a40e7b93b7f852b76e7d0 P ab704ad8154f4a4590e214cf9c1d7c4b: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20250905 19:57:49.893653 26888 catalog_manager.cc:5582] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5dbf236134d43158377e0310cee23d7 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "a5dbf236134d43158377e0310cee23d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } health_report { overall_health: UNKNOWN } } }
I20250905 19:57:49.906528 27243 raft_consensus.cc:491] T dbaec702dd3f48f2a1fcf12f8befed7e P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:57:49.907150 27243 raft_consensus.cc:513] T dbaec702dd3f48f2a1fcf12f8befed7e P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.909606 27243 leader_election.cc:290] T dbaec702dd3f48f2a1fcf12f8befed7e P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:57:49.911554 27141 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbaec702dd3f48f2a1fcf12f8befed7e" candidate_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" is_pre_election: true
I20250905 19:57:49.911990 27068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbaec702dd3f48f2a1fcf12f8befed7e" candidate_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5dbf236134d43158377e0310cee23d7" is_pre_election: true
I20250905 19:57:49.912793 27068 raft_consensus.cc:2391] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab704ad8154f4a4590e214cf9c1d7c4b in current term 1: Already voted for candidate ac86c30e5d044b9caf64b0c44f5cf7c8 in this term.
I20250905 19:57:49.913776 26957 leader_election.cc:304] T dbaec702dd3f48f2a1fcf12f8befed7e P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab704ad8154f4a4590e214cf9c1d7c4b; no voters: a5dbf236134d43158377e0310cee23d7, ac86c30e5d044b9caf64b0c44f5cf7c8
I20250905 19:57:49.914498 27243 raft_consensus.cc:3058] T dbaec702dd3f48f2a1fcf12f8befed7e P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:57:49.922468 27243 raft_consensus.cc:2747] T dbaec702dd3f48f2a1fcf12f8befed7e P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:57:49.944514 27216 consensus_queue.cc:1035] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:57:49.950632 27068 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7640f63d346f4d078ecb8d0e8a6c5670"
dest_uuid: "a5dbf236134d43158377e0310cee23d7"
mode: GRACEFUL
new_leader_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
from {username='slave'} at 127.0.0.1:44856
I20250905 19:57:49.951401 27068 raft_consensus.cc:604] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 1 LEADER]: Received request to transfer leadership to TS ab704ad8154f4a4590e214cf9c1d7c4b
I20250905 19:57:49.952441 27243 raft_consensus.cc:491] T 71b161547e6a40e7b93b7f852b76e7d0 P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:57:49.952953 27243 raft_consensus.cc:513] T 71b161547e6a40e7b93b7f852b76e7d0 P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.956176 27068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71b161547e6a40e7b93b7f852b76e7d0" candidate_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5dbf236134d43158377e0310cee23d7" is_pre_election: true
I20250905 19:57:49.959640 27141 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71b161547e6a40e7b93b7f852b76e7d0" candidate_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" is_pre_election: true
I20250905 19:57:49.960601 27243 leader_election.cc:290] T 71b161547e6a40e7b93b7f852b76e7d0 P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:57:49.963546 27255 consensus_queue.cc:1035] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:57:49.969318 26958 leader_election.cc:304] T 71b161547e6a40e7b93b7f852b76e7d0 P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab704ad8154f4a4590e214cf9c1d7c4b; no voters: a5dbf236134d43158377e0310cee23d7, ac86c30e5d044b9caf64b0c44f5cf7c8
I20250905 19:57:49.970283 27243 raft_consensus.cc:2747] T 71b161547e6a40e7b93b7f852b76e7d0 P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:57:49.980052 27256 raft_consensus.cc:491] T f6cfad67eeab45ddb61ef84d09f09b6b P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:57:49.980650 27256 raft_consensus.cc:513] T f6cfad67eeab45ddb61ef84d09f09b6b P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:49.983002 27256 leader_election.cc:290] T f6cfad67eeab45ddb61ef84d09f09b6b P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:57:49.985239 27141 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6cfad67eeab45ddb61ef84d09f09b6b" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" is_pre_election: true
I20250905 19:57:49.987787 27141 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c6a00cc443b94edcb46fbfdf1d7ac32c"
dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
mode: GRACEFUL
new_leader_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
from {username='slave'} at 127.0.0.1:60420
I20250905 19:57:49.988427 27141 raft_consensus.cc:604] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 LEADER]: Received request to transfer leadership to TS ab704ad8154f4a4590e214cf9c1d7c4b
I20250905 19:57:49.992703 26993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6cfad67eeab45ddb61ef84d09f09b6b" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" is_pre_election: true
I20250905 19:57:49.993722 26993 raft_consensus.cc:2391] T f6cfad67eeab45ddb61ef84d09f09b6b P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5dbf236134d43158377e0310cee23d7 in current term 1: Already voted for candidate ac86c30e5d044b9caf64b0c44f5cf7c8 in this term.
I20250905 19:57:49.995328 27033 leader_election.cc:304] T f6cfad67eeab45ddb61ef84d09f09b6b P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5dbf236134d43158377e0310cee23d7; no voters: ab704ad8154f4a4590e214cf9c1d7c4b, ac86c30e5d044b9caf64b0c44f5cf7c8
I20250905 19:57:49.995388 27141 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2f9344374d5649489c2f4336e9a1b15b"
dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
mode: GRACEFUL
new_leader_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
from {username='slave'} at 127.0.0.1:60420
I20250905 19:57:49.996060 27141 raft_consensus.cc:604] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 LEADER]: Received request to transfer leadership to TS ab704ad8154f4a4590e214cf9c1d7c4b
I20250905 19:57:49.996125 27216 raft_consensus.cc:3058] T f6cfad67eeab45ddb61ef84d09f09b6b P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:57:50.002184 27216 raft_consensus.cc:2747] T f6cfad67eeab45ddb61ef84d09f09b6b P a5dbf236134d43158377e0310cee23d7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:57:50.007974 27141 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "dbaec702dd3f48f2a1fcf12f8befed7e"
dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
mode: GRACEFUL
new_leader_uuid: "a5dbf236134d43158377e0310cee23d7"
from {username='slave'} at 127.0.0.1:60420
I20250905 19:57:50.008682 27141 raft_consensus.cc:604] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 LEADER]: Received request to transfer leadership to TS a5dbf236134d43158377e0310cee23d7
I20250905 19:57:50.023726 26993 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "685a60cc8a6841c6808fcba893636187"
dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
mode: GRACEFUL
new_leader_uuid: "a5dbf236134d43158377e0310cee23d7"
from {username='slave'} at 127.0.0.1:48440
I20250905 19:57:50.024397 26993 raft_consensus.cc:604] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 LEADER]: Received request to transfer leadership to TS a5dbf236134d43158377e0310cee23d7
I20250905 19:57:50.040148 27193 consensus_queue.cc:1035] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:57:50.047381 27232 raft_consensus.cc:991] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8: : Instructing follower a5dbf236134d43158377e0310cee23d7 to start an election
I20250905 19:57:50.048130 27236 raft_consensus.cc:1079] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 LEADER]: Signalling peer a5dbf236134d43158377e0310cee23d7 to start an election
I20250905 19:57:50.050500 27067 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "dbaec702dd3f48f2a1fcf12f8befed7e"
dest_uuid: "a5dbf236134d43158377e0310cee23d7"
from {username='slave'} at 127.0.0.1:44838
I20250905 19:57:50.051119 27067 raft_consensus.cc:491] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:57:50.051486 27067 raft_consensus.cc:3058] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:57:50.053067 27193 consensus_queue.cc:1035] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:57:50.058957 27067 raft_consensus.cc:513] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:50.060122 27068 raft_consensus.cc:1238] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Rejecting Update request from peer ac86c30e5d044b9caf64b0c44f5cf7c8 for earlier term 1. Current term is 2. Ops: []
I20250905 19:57:50.064415 27232 consensus_queue.cc:1046] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:57:50.064227 27236 consensus_queue.cc:1035] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:57:50.071182 27232 raft_consensus.cc:3053] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:57:50.071625 27232 raft_consensus.cc:738] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac86c30e5d044b9caf64b0c44f5cf7c8, State: Running, Role: LEADER
I20250905 19:57:50.072399 27232 consensus_queue.cc:260] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:50.073415 27232 raft_consensus.cc:3058] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:57:50.075431 27067 leader_election.cc:290] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 2 election: Requested vote from peers ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:57:50.075856 27193 raft_consensus.cc:991] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8: : Instructing follower ab704ad8154f4a4590e214cf9c1d7c4b to start an election
I20250905 19:57:50.076280 27193 raft_consensus.cc:1079] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 LEADER]: Signalling peer ab704ad8154f4a4590e214cf9c1d7c4b to start an election
I20250905 19:57:50.078938 27141 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbaec702dd3f48f2a1fcf12f8befed7e" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
I20250905 19:57:50.079666 26992 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "c6a00cc443b94edcb46fbfdf1d7ac32c"
dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
from {username='slave'} at 127.0.0.1:48422
I20250905 19:57:50.079996 26991 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbaec702dd3f48f2a1fcf12f8befed7e" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
I20250905 19:57:50.080248 26992 raft_consensus.cc:491] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:57:50.080602 26991 raft_consensus.cc:3058] T dbaec702dd3f48f2a1fcf12f8befed7e P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:57:50.080641 26992 raft_consensus.cc:3058] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 FOLLOWER]: Advancing to term 2
W20250905 19:57:50.078856 27236 consensus_queue.cc:1162] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20250905 19:57:50.097589 27232 consensus_queue.cc:1035] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:57:50.101752 27141 raft_consensus.cc:2466] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5dbf236134d43158377e0310cee23d7 in term 2.
I20250905 19:57:50.095831 26992 raft_consensus.cc:513] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:50.102109 26991 raft_consensus.cc:2466] T dbaec702dd3f48f2a1fcf12f8befed7e P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5dbf236134d43158377e0310cee23d7 in term 2.
I20250905 19:57:50.105418 26993 raft_consensus.cc:1238] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 FOLLOWER]: Rejecting Update request from peer ac86c30e5d044b9caf64b0c44f5cf7c8 for earlier term 1. Current term is 2. Ops: []
I20250905 19:57:50.107013 27033 leader_election.cc:304] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5dbf236134d43158377e0310cee23d7, ab704ad8154f4a4590e214cf9c1d7c4b; no voters:
I20250905 19:57:50.107964 27232 consensus_queue.cc:1046] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:57:50.108189 27216 raft_consensus.cc:2802] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:57:50.108857 27216 raft_consensus.cc:695] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 2 LEADER]: Becoming Leader. State: Replica: a5dbf236134d43158377e0310cee23d7, State: Running, Role: LEADER
I20250905 19:57:50.109717 27216 consensus_queue.cc:237] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:50.113173 27236 raft_consensus.cc:3053] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:57:50.113535 27236 raft_consensus.cc:738] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac86c30e5d044b9caf64b0c44f5cf7c8, State: Running, Role: LEADER
I20250905 19:57:50.114323 27236 consensus_queue.cc:260] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:50.115301 27236 raft_consensus.cc:3058] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:57:50.120704 26992 leader_election.cc:290] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 2 election: Requested vote from peers a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:57:50.121881 27067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6a00cc443b94edcb46fbfdf1d7ac32c" candidate_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a5dbf236134d43158377e0310cee23d7"
I20250905 19:57:50.122578 27067 raft_consensus.cc:3058] T c6a00cc443b94edcb46fbfdf1d7ac32c P a5dbf236134d43158377e0310cee23d7 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:57:50.123221 27141 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6a00cc443b94edcb46fbfdf1d7ac32c" candidate_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
I20250905 19:57:50.124990 26887 catalog_manager.cc:5582] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 reported cstate change: term changed from 1 to 2, leader changed from ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131) to a5dbf236134d43158377e0310cee23d7 (127.26.58.130). New cstate: current_term: 2 leader_uuid: "a5dbf236134d43158377e0310cee23d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } health_report { overall_health: UNKNOWN } } }
I20250905 19:57:50.132171 27067 raft_consensus.cc:2466] T c6a00cc443b94edcb46fbfdf1d7ac32c P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab704ad8154f4a4590e214cf9c1d7c4b in term 2.
I20250905 19:57:50.133404 26957 leader_election.cc:304] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5dbf236134d43158377e0310cee23d7, ab704ad8154f4a4590e214cf9c1d7c4b; no voters:
I20250905 19:57:50.133540 27141 raft_consensus.cc:2466] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab704ad8154f4a4590e214cf9c1d7c4b in term 2.
I20250905 19:57:50.134269 27243 raft_consensus.cc:2802] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:57:50.134770 27243 raft_consensus.cc:695] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 LEADER]: Becoming Leader. State: Replica: ab704ad8154f4a4590e214cf9c1d7c4b, State: Running, Role: LEADER
I20250905 19:57:50.135562 27243 consensus_queue.cc:237] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:50.141924 26887 catalog_manager.cc:5582] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b reported cstate change: term changed from 1 to 2, leader changed from ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131) to ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129). New cstate: current_term: 2 leader_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } health_report { overall_health: UNKNOWN } } }
I20250905 19:57:50.197844 27255 consensus_queue.cc:1035] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:57:50.204721 27216 raft_consensus.cc:991] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7: : Instructing follower ab704ad8154f4a4590e214cf9c1d7c4b to start an election
I20250905 19:57:50.205214 27216 raft_consensus.cc:1079] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 1 LEADER]: Signalling peer ab704ad8154f4a4590e214cf9c1d7c4b to start an election
I20250905 19:57:50.207511 26991 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7640f63d346f4d078ecb8d0e8a6c5670"
dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
from {username='slave'} at 127.0.0.1:48428
I20250905 19:57:50.208151 26991 raft_consensus.cc:491] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:57:50.208504 26991 raft_consensus.cc:3058] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:57:50.211066 27255 consensus_queue.cc:1035] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:57:50.216579 26991 raft_consensus.cc:513] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:50.218226 26992 raft_consensus.cc:1238] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 FOLLOWER]: Rejecting Update request from peer a5dbf236134d43158377e0310cee23d7 for earlier term 1. Current term is 2. Ops: []
I20250905 19:57:50.220932 27255 consensus_queue.cc:1046] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:57:50.222378 27216 raft_consensus.cc:3053] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:57:50.222487 27067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7640f63d346f4d078ecb8d0e8a6c5670" candidate_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a5dbf236134d43158377e0310cee23d7"
I20250905 19:57:50.222815 27216 raft_consensus.cc:738] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a5dbf236134d43158377e0310cee23d7, State: Running, Role: LEADER
I20250905 19:57:50.223624 27216 consensus_queue.cc:260] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:50.224157 26991 leader_election.cc:290] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 2 election: Requested vote from peers a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:57:50.224536 27216 raft_consensus.cc:3058] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:57:50.226821 27141 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7640f63d346f4d078ecb8d0e8a6c5670" candidate_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
I20250905 19:57:50.227386 27141 raft_consensus.cc:3058] T 7640f63d346f4d078ecb8d0e8a6c5670 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:57:50.234151 27141 raft_consensus.cc:2466] T 7640f63d346f4d078ecb8d0e8a6c5670 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab704ad8154f4a4590e214cf9c1d7c4b in term 2.
I20250905 19:57:50.235242 26958 leader_election.cc:304] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab704ad8154f4a4590e214cf9c1d7c4b, ac86c30e5d044b9caf64b0c44f5cf7c8; no voters:
I20250905 19:57:50.235922 27243 raft_consensus.cc:2802] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:57:50.236311 27243 raft_consensus.cc:695] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 LEADER]: Becoming Leader. State: Replica: ab704ad8154f4a4590e214cf9c1d7c4b, State: Running, Role: LEADER
I20250905 19:57:50.237138 27243 consensus_queue.cc:237] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:50.238283 27067 raft_consensus.cc:2466] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab704ad8154f4a4590e214cf9c1d7c4b in term 2.
I20250905 19:57:50.244709 26887 catalog_manager.cc:5582] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b reported cstate change: term changed from 1 to 2, leader changed from a5dbf236134d43158377e0310cee23d7 (127.26.58.130) to ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129). New cstate: current_term: 2 leader_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } health_report { overall_health: UNKNOWN } } }
I20250905 19:57:50.246320 27190 raft_consensus.cc:991] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b: : Instructing follower a5dbf236134d43158377e0310cee23d7 to start an election
I20250905 19:57:50.246095 27232 consensus_queue.cc:1035] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:57:50.246796 27190 raft_consensus.cc:1079] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 LEADER]: Signalling peer a5dbf236134d43158377e0310cee23d7 to start an election
I20250905 19:57:50.248914 27066 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "685a60cc8a6841c6808fcba893636187"
dest_uuid: "a5dbf236134d43158377e0310cee23d7"
from {username='slave'} at 127.0.0.1:44842
I20250905 19:57:50.250550 27066 raft_consensus.cc:491] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:57:50.251058 27066 raft_consensus.cc:3058] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:57:50.258524 27236 consensus_queue.cc:1035] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:57:50.263098 27216 raft_consensus.cc:491] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:57:50.263633 27216 raft_consensus.cc:513] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:50.265570 27216 leader_election.cc:290] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:57:50.266525 27066 raft_consensus.cc:513] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:50.268568 27066 leader_election.cc:290] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 2 election: Requested vote from peers ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:57:50.272008 26992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c08349e8a8394ee182d5d0928074181f" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" is_pre_election: true
I20250905 19:57:50.274356 27141 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c08349e8a8394ee182d5d0928074181f" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" is_pre_election: true
I20250905 19:57:50.275315 27141 raft_consensus.cc:2391] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5dbf236134d43158377e0310cee23d7 in current term 1: Already voted for candidate ab704ad8154f4a4590e214cf9c1d7c4b in this term.
I20250905 19:57:50.275189 26991 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "685a60cc8a6841c6808fcba893636187" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
I20250905 19:57:50.275779 26991 raft_consensus.cc:3053] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:57:50.276137 26991 raft_consensus.cc:738] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab704ad8154f4a4590e214cf9c1d7c4b, State: Running, Role: LEADER
I20250905 19:57:50.276031 27193 consensus_queue.cc:1035] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:57:50.274384 27142 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "685a60cc8a6841c6808fcba893636187" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
I20250905 19:57:50.277776 27142 raft_consensus.cc:3058] T 685a60cc8a6841c6808fcba893636187 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:57:50.280889 27033 leader_election.cc:304] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5dbf236134d43158377e0310cee23d7; no voters: ab704ad8154f4a4590e214cf9c1d7c4b, ac86c30e5d044b9caf64b0c44f5cf7c8
I20250905 19:57:50.284973 27216 raft_consensus.cc:3058] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:57:50.288879 27193 consensus_queue.cc:1035] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:57:50.277042 26991 consensus_queue.cc:260] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:50.291093 26991 raft_consensus.cc:3058] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:57:50.312565 27142 raft_consensus.cc:2466] T 685a60cc8a6841c6808fcba893636187 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5dbf236134d43158377e0310cee23d7 in term 2.
I20250905 19:57:50.314309 27033 leader_election.cc:304] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5dbf236134d43158377e0310cee23d7, ac86c30e5d044b9caf64b0c44f5cf7c8; no voters:
I20250905 19:57:50.314929 26991 raft_consensus.cc:2466] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5dbf236134d43158377e0310cee23d7 in term 2.
I20250905 19:57:50.315135 27216 raft_consensus.cc:2747] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:57:50.316123 27255 raft_consensus.cc:2802] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:57:50.316623 27255 raft_consensus.cc:695] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7 [term 2 LEADER]: Becoming Leader. State: Replica: a5dbf236134d43158377e0310cee23d7, State: Running, Role: LEADER
I20250905 19:57:50.317442 27255 consensus_queue.cc:237] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:50.324883 27204 consensus_queue.cc:1035] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:57:50.333041 26992 raft_consensus.cc:3058] T 71b161547e6a40e7b93b7f852b76e7d0 P ab704ad8154f4a4590e214cf9c1d7c4b [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:57:50.338711 27204 consensus_queue.cc:1035] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:57:50.343019 26889 catalog_manager.cc:5582] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7 reported cstate change: term changed from 1 to 2, leader changed from ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129) to a5dbf236134d43158377e0310cee23d7 (127.26.58.130). New cstate: current_term: 2 leader_uuid: "a5dbf236134d43158377e0310cee23d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } health_report { overall_health: UNKNOWN } } }
I20250905 19:57:50.383046 27236 raft_consensus.cc:991] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8: : Instructing follower ab704ad8154f4a4590e214cf9c1d7c4b to start an election
I20250905 19:57:50.383664 27236 raft_consensus.cc:1079] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 LEADER]: Signalling peer ab704ad8154f4a4590e214cf9c1d7c4b to start an election
I20250905 19:57:50.386111 26991 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "2f9344374d5649489c2f4336e9a1b15b"
dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
from {username='slave'} at 127.0.0.1:48422
I20250905 19:57:50.386998 26991 raft_consensus.cc:491] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:57:50.387498 26991 raft_consensus.cc:3058] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:57:50.401747 26991 raft_consensus.cc:513] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:50.413374 26991 leader_election.cc:290] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 2 election: Requested vote from peers a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:57:50.415107 27067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f9344374d5649489c2f4336e9a1b15b" candidate_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a5dbf236134d43158377e0310cee23d7"
I20250905 19:57:50.415731 27067 raft_consensus.cc:3058] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:57:50.417613 27142 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f9344374d5649489c2f4336e9a1b15b" candidate_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
I20250905 19:57:50.418212 27142 raft_consensus.cc:3053] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:57:50.418563 27142 raft_consensus.cc:738] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac86c30e5d044b9caf64b0c44f5cf7c8, State: Running, Role: LEADER
I20250905 19:57:50.419478 27142 consensus_queue.cc:260] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:50.421109 27142 raft_consensus.cc:3058] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:57:50.422629 27067 raft_consensus.cc:2466] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab704ad8154f4a4590e214cf9c1d7c4b in term 2.
I20250905 19:57:50.423916 26957 leader_election.cc:304] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5dbf236134d43158377e0310cee23d7, ab704ad8154f4a4590e214cf9c1d7c4b; no voters:
I20250905 19:57:50.424717 27204 raft_consensus.cc:2802] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:57:50.425197 27204 raft_consensus.cc:695] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 LEADER]: Becoming Leader. State: Replica: ab704ad8154f4a4590e214cf9c1d7c4b, State: Running, Role: LEADER
I20250905 19:57:50.426024 27204 consensus_queue.cc:237] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:50.429972 27142 raft_consensus.cc:2466] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab704ad8154f4a4590e214cf9c1d7c4b in term 2.
I20250905 19:57:50.442904 26889 catalog_manager.cc:5582] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b reported cstate change: term changed from 1 to 2, leader changed from ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131) to ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129). New cstate: current_term: 2 leader_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } health_report { overall_health: UNKNOWN } } }
I20250905 19:57:50.509171 26991 raft_consensus.cc:1273] T dbaec702dd3f48f2a1fcf12f8befed7e P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 FOLLOWER]: Refusing update from remote peer a5dbf236134d43158377e0310cee23d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:57:50.510466 27192 consensus_queue.cc:1035] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:57:50.520267 27142 raft_consensus.cc:1273] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 FOLLOWER]: Refusing update from remote peer a5dbf236134d43158377e0310cee23d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:57:50.522054 27192 consensus_queue.cc:1035] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:57:50.655129 27142 raft_consensus.cc:1273] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 FOLLOWER]: Refusing update from remote peer ab704ad8154f4a4590e214cf9c1d7c4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:57:50.656664 27204 consensus_queue.cc:1035] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:57:50.666810 27067 raft_consensus.cc:1273] T c6a00cc443b94edcb46fbfdf1d7ac32c P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Refusing update from remote peer ab704ad8154f4a4590e214cf9c1d7c4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:57:50.668193 27190 consensus_queue.cc:1035] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:57:50.704685 27067 raft_consensus.cc:1273] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Refusing update from remote peer ab704ad8154f4a4590e214cf9c1d7c4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:57:50.706103 27190 consensus_queue.cc:1035] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:57:50.716302 27142 raft_consensus.cc:1273] T 7640f63d346f4d078ecb8d0e8a6c5670 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 FOLLOWER]: Refusing update from remote peer ab704ad8154f4a4590e214cf9c1d7c4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:57:50.718164 27245 consensus_queue.cc:1035] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:57:50.787827 27142 raft_consensus.cc:1273] T 685a60cc8a6841c6808fcba893636187 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 FOLLOWER]: Refusing update from remote peer a5dbf236134d43158377e0310cee23d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:57:50.789363 27192 consensus_queue.cc:1035] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:57:50.798205 26991 raft_consensus.cc:1273] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 FOLLOWER]: Refusing update from remote peer a5dbf236134d43158377e0310cee23d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:57:50.799978 27250 consensus_queue.cc:1035] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:57:50.962163 27142 raft_consensus.cc:1273] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 FOLLOWER]: Refusing update from remote peer ab704ad8154f4a4590e214cf9c1d7c4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:57:50.963404 27245 consensus_queue.cc:1035] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:57:50.971858 27067 raft_consensus.cc:1273] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Refusing update from remote peer ab704ad8154f4a4590e214cf9c1d7c4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:57:50.973627 27245 consensus_queue.cc:1035] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:57:53.054952 26858 auto_leader_rebalancer-test.cc:272] The leader distribution is
I20250905 19:57:53.055743 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:57:53.071893 26992 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c08349e8a8394ee182d5d0928074181f"
dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
mode: GRACEFUL
new_leader_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
from {username='slave'} at 127.0.0.1:51378
I20250905 19:57:53.072409 26992 raft_consensus.cc:604] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 LEADER]: Received request to transfer leadership to TS ac86c30e5d044b9caf64b0c44f5cf7c8
I20250905 19:57:53.082188 27067 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "dbaec702dd3f48f2a1fcf12f8befed7e"
dest_uuid: "a5dbf236134d43158377e0310cee23d7"
mode: GRACEFUL
new_leader_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
from {username='slave'} at 127.0.0.1:46142
I20250905 19:57:53.082871 27067 raft_consensus.cc:604] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 2 LEADER]: Received request to transfer leadership to TS ac86c30e5d044b9caf64b0c44f5cf7c8
I20250905 19:57:53.083952 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250905 19:57:53.084393 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:57:53.270107 27250 raft_consensus.cc:991] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7: : Instructing follower ac86c30e5d044b9caf64b0c44f5cf7c8 to start an election
I20250905 19:57:53.270637 27278 raft_consensus.cc:1079] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 2 LEADER]: Signalling peer ac86c30e5d044b9caf64b0c44f5cf7c8 to start an election
I20250905 19:57:53.272112 27141 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "dbaec702dd3f48f2a1fcf12f8befed7e"
dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
from {username='slave'} at 127.0.0.1:60404
I20250905 19:57:53.272584 27141 raft_consensus.cc:491] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:57:53.272869 27141 raft_consensus.cc:3058] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:57:53.305485 27141 raft_consensus.cc:513] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:53.307546 27141 leader_election.cc:290] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 3 election: Requested vote from peers ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107), a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875)
I20250905 19:57:53.308521 26992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbaec702dd3f48f2a1fcf12f8befed7e" candidate_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
I20250905 19:57:53.308609 27067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbaec702dd3f48f2a1fcf12f8befed7e" candidate_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a5dbf236134d43158377e0310cee23d7"
I20250905 19:57:53.309180 26992 raft_consensus.cc:3058] T dbaec702dd3f48f2a1fcf12f8befed7e P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:57:53.309238 27067 raft_consensus.cc:3053] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:57:53.309589 27067 raft_consensus.cc:738] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a5dbf236134d43158377e0310cee23d7, State: Running, Role: LEADER
I20250905 19:57:53.310236 27067 consensus_queue.cc:260] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:53.311244 27067 raft_consensus.cc:3058] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:57:53.314843 26992 raft_consensus.cc:2466] T dbaec702dd3f48f2a1fcf12f8befed7e P ab704ad8154f4a4590e214cf9c1d7c4b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac86c30e5d044b9caf64b0c44f5cf7c8 in term 3.
I20250905 19:57:53.316179 27067 raft_consensus.cc:2466] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac86c30e5d044b9caf64b0c44f5cf7c8 in term 3.
I20250905 19:57:53.316291 27107 leader_election.cc:304] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab704ad8154f4a4590e214cf9c1d7c4b, ac86c30e5d044b9caf64b0c44f5cf7c8; no voters:
I20250905 19:57:53.317317 27193 raft_consensus.cc:2802] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:57:53.318524 27193 raft_consensus.cc:695] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 3 LEADER]: Becoming Leader. State: Replica: ac86c30e5d044b9caf64b0c44f5cf7c8, State: Running, Role: LEADER
I20250905 19:57:53.319254 27193 consensus_queue.cc:237] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:53.328333 26889 catalog_manager.cc:5582] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 reported cstate change: term changed from 2 to 3, leader changed from a5dbf236134d43158377e0310cee23d7 (127.26.58.130) to ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131). New cstate: current_term: 3 leader_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } health_report { overall_health: HEALTHY } } }
I20250905 19:57:53.604871 27190 raft_consensus.cc:991] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b: : Instructing follower ac86c30e5d044b9caf64b0c44f5cf7c8 to start an election
I20250905 19:57:53.605243 27288 raft_consensus.cc:1079] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 LEADER]: Signalling peer ac86c30e5d044b9caf64b0c44f5cf7c8 to start an election
I20250905 19:57:53.606575 27141 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "c08349e8a8394ee182d5d0928074181f"
dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
from {username='slave'} at 127.0.0.1:60388
I20250905 19:57:53.607035 27141 raft_consensus.cc:491] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:57:53.607321 27141 raft_consensus.cc:3058] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:57:53.621248 27141 raft_consensus.cc:513] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:53.622933 27141 leader_election.cc:290] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 2 election: Requested vote from peers a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875), ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107)
I20250905 19:57:53.624893 26992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c08349e8a8394ee182d5d0928074181f" candidate_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
I20250905 19:57:53.624994 27067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c08349e8a8394ee182d5d0928074181f" candidate_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a5dbf236134d43158377e0310cee23d7"
I20250905 19:57:53.625691 26992 raft_consensus.cc:3053] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:57:53.625762 27067 raft_consensus.cc:3058] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:57:53.626189 26992 raft_consensus.cc:738] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab704ad8154f4a4590e214cf9c1d7c4b, State: Running, Role: LEADER
I20250905 19:57:53.626991 26992 consensus_queue.cc:260] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:53.627964 26992 raft_consensus.cc:3058] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:57:53.718750 27067 raft_consensus.cc:2466] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac86c30e5d044b9caf64b0c44f5cf7c8 in term 2.
I20250905 19:57:53.718748 26992 raft_consensus.cc:2466] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac86c30e5d044b9caf64b0c44f5cf7c8 in term 2.
I20250905 19:57:53.720216 27106 leader_election.cc:304] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5dbf236134d43158377e0310cee23d7, ac86c30e5d044b9caf64b0c44f5cf7c8; no voters:
I20250905 19:57:53.721135 27193 raft_consensus.cc:2802] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:57:53.721642 27193 raft_consensus.cc:695] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 LEADER]: Becoming Leader. State: Replica: ac86c30e5d044b9caf64b0c44f5cf7c8, State: Running, Role: LEADER
I20250905 19:57:53.722398 27193 consensus_queue.cc:237] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:57:53.729420 26889 catalog_manager.cc:5582] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 reported cstate change: term changed from 1 to 2, leader changed from ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129) to ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131). New cstate: current_term: 2 leader_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } health_report { overall_health: HEALTHY } } }
I20250905 19:57:53.733206 27067 raft_consensus.cc:1273] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 3 FOLLOWER]: Refusing update from remote peer ac86c30e5d044b9caf64b0c44f5cf7c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:57:53.734503 27193 consensus_queue.cc:1035] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:57:53.744412 26992 raft_consensus.cc:1273] T dbaec702dd3f48f2a1fcf12f8befed7e P ab704ad8154f4a4590e214cf9c1d7c4b [term 3 FOLLOWER]: Refusing update from remote peer ac86c30e5d044b9caf64b0c44f5cf7c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:57:53.745678 27193 consensus_queue.cc:1035] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:57:54.085315 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:57:54.090174 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:57:54.090569 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:57:54.203199 26992 raft_consensus.cc:1273] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 FOLLOWER]: Refusing update from remote peer ac86c30e5d044b9caf64b0c44f5cf7c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:57:54.204319 27295 consensus_queue.cc:1035] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:57:54.212600 27067 raft_consensus.cc:1273] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Refusing update from remote peer ac86c30e5d044b9caf64b0c44f5cf7c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:57:54.213858 27295 consensus_queue.cc:1035] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:57:55.091358 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:57:55.096880 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:57:55.097271 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:57:56.098531 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:57:56.103811 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:57:56.104167 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:57:57.105152 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:57:57.109794 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:57:57.110170 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:57:58.111033 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:57:58.116951 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:57:58.117341 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:57:59.302249 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:57:59.307143 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:57:59.307478 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250905 19:57:59.718837 27163 debug-util.cc:398] Leaking SignalData structure 0x7b080016c060 after lost signal to thread 26861
W20250905 19:57:59.719612 27163 debug-util.cc:398] Leaking SignalData structure 0x7b080016c740 after lost signal to thread 26924
W20250905 19:57:59.720353 27163 debug-util.cc:398] Leaking SignalData structure 0x7b0800177280 after lost signal to thread 27017
W20250905 19:57:59.721161 27163 debug-util.cc:398] Leaking SignalData structure 0x7b080018b080 after lost signal to thread 27092
W20250905 19:57:59.721966 27163 debug-util.cc:398] Leaking SignalData structure 0x7b0800177a20 after lost signal to thread 27166
I20250905 19:58:00.308315 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:00.313094 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:00.313521 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:01.314487 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:01.320708 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:01.321111 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:02.322018 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:02.329206 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:02.329699 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:03.356559 26858 auto_leader_rebalancer-test.cc:291] The leader distribution is
I20250905 19:58:03.361606 27141 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c08349e8a8394ee182d5d0928074181f"
dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
mode: GRACEFUL
new_leader_uuid: "a5dbf236134d43158377e0310cee23d7"
from {username='slave'} at 127.0.0.1:60420
I20250905 19:58:03.362275 27141 raft_consensus.cc:604] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 LEADER]: Received request to transfer leadership to TS a5dbf236134d43158377e0310cee23d7
I20250905 19:58:03.367071 26992 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7640f63d346f4d078ecb8d0e8a6c5670"
dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
mode: GRACEFUL
new_leader_uuid: "a5dbf236134d43158377e0310cee23d7"
from {username='slave'} at 127.0.0.1:48440
I20250905 19:58:03.367523 26992 raft_consensus.cc:604] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 LEADER]: Received request to transfer leadership to TS a5dbf236134d43158377e0310cee23d7
I20250905 19:58:03.372031 26992 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c6a00cc443b94edcb46fbfdf1d7ac32c"
dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
mode: GRACEFUL
new_leader_uuid: "a5dbf236134d43158377e0310cee23d7"
from {username='slave'} at 127.0.0.1:48440
I20250905 19:58:03.372457 26992 raft_consensus.cc:604] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 LEADER]: Received request to transfer leadership to TS a5dbf236134d43158377e0310cee23d7
I20250905 19:58:03.374418 27308 raft_consensus.cc:991] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b: : Instructing follower a5dbf236134d43158377e0310cee23d7 to start an election
I20250905 19:58:03.374989 27288 raft_consensus.cc:1079] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 LEADER]: Signalling peer a5dbf236134d43158377e0310cee23d7 to start an election
I20250905 19:58:03.376508 27066 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "c6a00cc443b94edcb46fbfdf1d7ac32c"
dest_uuid: "a5dbf236134d43158377e0310cee23d7"
from {username='slave'} at 127.0.0.1:44842
I20250905 19:58:03.377009 26992 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2f9344374d5649489c2f4336e9a1b15b"
dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
mode: GRACEFUL
new_leader_uuid: "a5dbf236134d43158377e0310cee23d7"
from {username='slave'} at 127.0.0.1:48440
I20250905 19:58:03.377056 27066 raft_consensus.cc:491] T c6a00cc443b94edcb46fbfdf1d7ac32c P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:58:03.377507 26992 raft_consensus.cc:604] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 LEADER]: Received request to transfer leadership to TS a5dbf236134d43158377e0310cee23d7
I20250905 19:58:03.377552 27066 raft_consensus.cc:3058] T c6a00cc443b94edcb46fbfdf1d7ac32c P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:03.384351 27066 raft_consensus.cc:513] T c6a00cc443b94edcb46fbfdf1d7ac32c P a5dbf236134d43158377e0310cee23d7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:03.386550 27066 leader_election.cc:290] T c6a00cc443b94edcb46fbfdf1d7ac32c P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 3 election: Requested vote from peers ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:58:03.387157 26992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6a00cc443b94edcb46fbfdf1d7ac32c" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
I20250905 19:58:03.387459 27141 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "dbaec702dd3f48f2a1fcf12f8befed7e"
dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
mode: GRACEFUL
new_leader_uuid: "a5dbf236134d43158377e0310cee23d7"
from {username='slave'} at 127.0.0.1:60420
I20250905 19:58:03.387961 27141 raft_consensus.cc:604] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 3 LEADER]: Received request to transfer leadership to TS a5dbf236134d43158377e0310cee23d7
I20250905 19:58:03.387945 26992 raft_consensus.cc:3053] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:58:03.388480 26992 raft_consensus.cc:738] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ab704ad8154f4a4590e214cf9c1d7c4b, State: Running, Role: LEADER
I20250905 19:58:03.388545 27142 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6a00cc443b94edcb46fbfdf1d7ac32c" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
I20250905 19:58:03.389135 27142 raft_consensus.cc:3058] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:03.389307 26992 consensus_queue.cc:260] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:03.390616 26992 raft_consensus.cc:3058] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:03.397022 26992 raft_consensus.cc:2466] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5dbf236134d43158377e0310cee23d7 in term 3.
I20250905 19:58:03.398623 27142 raft_consensus.cc:2466] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5dbf236134d43158377e0310cee23d7 in term 3.
I20250905 19:58:03.398581 27033 leader_election.cc:304] T c6a00cc443b94edcb46fbfdf1d7ac32c P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5dbf236134d43158377e0310cee23d7, ab704ad8154f4a4590e214cf9c1d7c4b; no voters:
I20250905 19:58:03.398922 27308 raft_consensus.cc:991] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b: : Instructing follower a5dbf236134d43158377e0310cee23d7 to start an election
I20250905 19:58:03.399513 27288 raft_consensus.cc:1079] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 LEADER]: Signalling peer a5dbf236134d43158377e0310cee23d7 to start an election
I20250905 19:58:03.399695 27302 raft_consensus.cc:2802] T c6a00cc443b94edcb46fbfdf1d7ac32c P a5dbf236134d43158377e0310cee23d7 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:58:03.400741 27302 raft_consensus.cc:695] T c6a00cc443b94edcb46fbfdf1d7ac32c P a5dbf236134d43158377e0310cee23d7 [term 3 LEADER]: Becoming Leader. State: Replica: a5dbf236134d43158377e0310cee23d7, State: Running, Role: LEADER
I20250905 19:58:03.401796 27066 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "2f9344374d5649489c2f4336e9a1b15b"
dest_uuid: "a5dbf236134d43158377e0310cee23d7"
from {username='slave'} at 127.0.0.1:44842
I20250905 19:58:03.401681 27302 consensus_queue.cc:237] T c6a00cc443b94edcb46fbfdf1d7ac32c P a5dbf236134d43158377e0310cee23d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:03.402335 27066 raft_consensus.cc:491] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:58:03.402730 27066 raft_consensus.cc:3058] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:03.408452 27066 raft_consensus.cc:513] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:03.411857 27066 leader_election.cc:290] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 3 election: Requested vote from peers ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:58:03.412797 26992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f9344374d5649489c2f4336e9a1b15b" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
I20250905 19:58:03.413434 27142 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f9344374d5649489c2f4336e9a1b15b" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
I20250905 19:58:03.413533 26992 raft_consensus.cc:3053] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:58:03.413885 27142 raft_consensus.cc:3058] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:03.413913 26992 raft_consensus.cc:738] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ab704ad8154f4a4590e214cf9c1d7c4b, State: Running, Role: LEADER
I20250905 19:58:03.413787 26889 catalog_manager.cc:5582] T c6a00cc443b94edcb46fbfdf1d7ac32c P a5dbf236134d43158377e0310cee23d7 reported cstate change: term changed from 2 to 3, leader changed from ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129) to a5dbf236134d43158377e0310cee23d7 (127.26.58.130). New cstate: current_term: 3 leader_uuid: "a5dbf236134d43158377e0310cee23d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:03.414727 26992 consensus_queue.cc:260] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:03.415931 26992 raft_consensus.cc:3058] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:03.421538 27142 raft_consensus.cc:2466] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5dbf236134d43158377e0310cee23d7 in term 3.
I20250905 19:58:03.423055 26992 raft_consensus.cc:2466] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5dbf236134d43158377e0310cee23d7 in term 3.
I20250905 19:58:03.424058 27033 leader_election.cc:304] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5dbf236134d43158377e0310cee23d7, ab704ad8154f4a4590e214cf9c1d7c4b; no voters:
I20250905 19:58:03.424896 27302 raft_consensus.cc:2802] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:58:03.425426 27302 raft_consensus.cc:695] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [term 3 LEADER]: Becoming Leader. State: Replica: a5dbf236134d43158377e0310cee23d7, State: Running, Role: LEADER
I20250905 19:58:03.426288 27302 consensus_queue.cc:237] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:03.428210 27311 raft_consensus.cc:991] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8: : Instructing follower a5dbf236134d43158377e0310cee23d7 to start an election
I20250905 19:58:03.428745 27305 raft_consensus.cc:1079] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 LEADER]: Signalling peer a5dbf236134d43158377e0310cee23d7 to start an election
I20250905 19:58:03.430888 27066 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "c08349e8a8394ee182d5d0928074181f"
dest_uuid: "a5dbf236134d43158377e0310cee23d7"
from {username='slave'} at 127.0.0.1:44838
I20250905 19:58:03.431599 27066 raft_consensus.cc:491] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:58:03.432073 27066 raft_consensus.cc:3058] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:03.443013 27066 raft_consensus.cc:513] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:03.444674 27066 leader_election.cc:290] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 3 election: Requested vote from peers ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:58:03.445760 26992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c08349e8a8394ee182d5d0928074181f" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
I20250905 19:58:03.446333 26992 raft_consensus.cc:3058] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:03.446245 27142 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c08349e8a8394ee182d5d0928074181f" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
I20250905 19:58:03.446987 27142 raft_consensus.cc:3053] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:58:03.447314 27142 raft_consensus.cc:738] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ac86c30e5d044b9caf64b0c44f5cf7c8, State: Running, Role: LEADER
I20250905 19:58:03.448171 27142 consensus_queue.cc:260] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:03.449401 27142 raft_consensus.cc:3058] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:03.451191 26992 raft_consensus.cc:2466] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5dbf236134d43158377e0310cee23d7 in term 3.
I20250905 19:58:03.452111 27033 leader_election.cc:304] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5dbf236134d43158377e0310cee23d7, ab704ad8154f4a4590e214cf9c1d7c4b; no voters:
I20250905 19:58:03.452823 27302 raft_consensus.cc:2802] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:58:03.453539 27302 raft_consensus.cc:695] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [term 3 LEADER]: Becoming Leader. State: Replica: a5dbf236134d43158377e0310cee23d7, State: Running, Role: LEADER
I20250905 19:58:03.454460 27302 consensus_queue.cc:237] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:03.454322 26889 catalog_manager.cc:5582] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 reported cstate change: term changed from 2 to 3, leader changed from ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129) to a5dbf236134d43158377e0310cee23d7 (127.26.58.130). New cstate: current_term: 3 leader_uuid: "a5dbf236134d43158377e0310cee23d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:03.457703 27142 raft_consensus.cc:2466] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5dbf236134d43158377e0310cee23d7 in term 3.
I20250905 19:58:03.465857 26889 catalog_manager.cc:5582] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 reported cstate change: term changed from 2 to 3, leader changed from ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131) to a5dbf236134d43158377e0310cee23d7 (127.26.58.130). New cstate: current_term: 3 leader_uuid: "a5dbf236134d43158377e0310cee23d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:03.547093 27288 raft_consensus.cc:991] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b: : Instructing follower a5dbf236134d43158377e0310cee23d7 to start an election
I20250905 19:58:03.547466 27308 raft_consensus.cc:1079] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 LEADER]: Signalling peer a5dbf236134d43158377e0310cee23d7 to start an election
I20250905 19:58:03.548801 27066 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7640f63d346f4d078ecb8d0e8a6c5670"
dest_uuid: "a5dbf236134d43158377e0310cee23d7"
from {username='slave'} at 127.0.0.1:44842
I20250905 19:58:03.549283 27066 raft_consensus.cc:491] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:58:03.549551 27066 raft_consensus.cc:3058] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:03.553773 27066 raft_consensus.cc:513] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:03.555337 27066 leader_election.cc:290] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 3 election: Requested vote from peers ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:58:03.556263 26992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7640f63d346f4d078ecb8d0e8a6c5670" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
I20250905 19:58:03.556295 27142 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7640f63d346f4d078ecb8d0e8a6c5670" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
I20250905 19:58:03.556739 26992 raft_consensus.cc:3053] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:58:03.556835 27142 raft_consensus.cc:3058] T 7640f63d346f4d078ecb8d0e8a6c5670 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:03.557003 26992 raft_consensus.cc:738] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ab704ad8154f4a4590e214cf9c1d7c4b, State: Running, Role: LEADER
I20250905 19:58:03.557559 26992 consensus_queue.cc:260] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:03.558363 26992 raft_consensus.cc:3058] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:03.561395 27142 raft_consensus.cc:2466] T 7640f63d346f4d078ecb8d0e8a6c5670 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5dbf236134d43158377e0310cee23d7 in term 3.
I20250905 19:58:03.562055 26992 raft_consensus.cc:2466] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5dbf236134d43158377e0310cee23d7 in term 3.
I20250905 19:58:03.562301 27033 leader_election.cc:304] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5dbf236134d43158377e0310cee23d7, ac86c30e5d044b9caf64b0c44f5cf7c8; no voters:
I20250905 19:58:03.562963 27302 raft_consensus.cc:2802] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:58:03.563333 27302 raft_consensus.cc:695] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 3 LEADER]: Becoming Leader. State: Replica: a5dbf236134d43158377e0310cee23d7, State: Running, Role: LEADER
I20250905 19:58:03.563934 27302 consensus_queue.cc:237] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:03.570742 26889 catalog_manager.cc:5582] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 reported cstate change: term changed from 2 to 3, leader changed from ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129) to a5dbf236134d43158377e0310cee23d7 (127.26.58.130). New cstate: current_term: 3 leader_uuid: "a5dbf236134d43158377e0310cee23d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:03.821642 26991 raft_consensus.cc:1273] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b [term 3 FOLLOWER]: Refusing update from remote peer a5dbf236134d43158377e0310cee23d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:58:03.822728 27302 consensus_queue.cc:1035] T c6a00cc443b94edcb46fbfdf1d7ac32c P a5dbf236134d43158377e0310cee23d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:58:03.831507 27142 raft_consensus.cc:1273] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 3 FOLLOWER]: Refusing update from remote peer a5dbf236134d43158377e0310cee23d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:58:03.833092 27302 consensus_queue.cc:1035] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:58:03.849006 27141 raft_consensus.cc:1273] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 3 FOLLOWER]: Refusing update from remote peer a5dbf236134d43158377e0310cee23d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:58:03.851197 26992 raft_consensus.cc:1273] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [term 3 FOLLOWER]: Refusing update from remote peer a5dbf236134d43158377e0310cee23d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:58:03.850868 27321 consensus_queue.cc:1035] T c6a00cc443b94edcb46fbfdf1d7ac32c P a5dbf236134d43158377e0310cee23d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:58:03.852627 27302 consensus_queue.cc:1035] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:58:03.867795 27305 raft_consensus.cc:991] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8: : Instructing follower a5dbf236134d43158377e0310cee23d7 to start an election
I20250905 19:58:03.868321 27313 raft_consensus.cc:1079] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 3 LEADER]: Signalling peer a5dbf236134d43158377e0310cee23d7 to start an election
I20250905 19:58:03.869760 27066 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "dbaec702dd3f48f2a1fcf12f8befed7e"
dest_uuid: "a5dbf236134d43158377e0310cee23d7"
from {username='slave'} at 127.0.0.1:44838
I20250905 19:58:03.870199 27066 raft_consensus.cc:491] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:58:03.870508 27066 raft_consensus.cc:3058] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:58:03.875341 27066 raft_consensus.cc:513] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:03.876974 27066 leader_election.cc:290] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 4 election: Requested vote from peers ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:58:03.877763 26992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbaec702dd3f48f2a1fcf12f8befed7e" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
I20250905 19:58:03.877943 27141 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbaec702dd3f48f2a1fcf12f8befed7e" candidate_uuid: "a5dbf236134d43158377e0310cee23d7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
I20250905 19:58:03.878479 27141 raft_consensus.cc:3053] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:58:03.878800 27141 raft_consensus.cc:738] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ac86c30e5d044b9caf64b0c44f5cf7c8, State: Running, Role: LEADER
I20250905 19:58:03.879801 27141 consensus_queue.cc:260] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:03.880645 27141 raft_consensus.cc:3058] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:58:03.884882 27141 raft_consensus.cc:2466] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5dbf236134d43158377e0310cee23d7 in term 4.
I20250905 19:58:03.885769 27033 leader_election.cc:304] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5dbf236134d43158377e0310cee23d7, ac86c30e5d044b9caf64b0c44f5cf7c8; no voters: ab704ad8154f4a4590e214cf9c1d7c4b
I20250905 19:58:03.886512 27309 raft_consensus.cc:2802] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:58:03.886932 27309 raft_consensus.cc:695] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 4 LEADER]: Becoming Leader. State: Replica: a5dbf236134d43158377e0310cee23d7, State: Running, Role: LEADER
I20250905 19:58:03.887583 27309 consensus_queue.cc:237] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:03.894353 26889 catalog_manager.cc:5582] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 reported cstate change: term changed from 3 to 4, leader changed from ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131) to a5dbf236134d43158377e0310cee23d7 (127.26.58.130). New cstate: current_term: 4 leader_uuid: "a5dbf236134d43158377e0310cee23d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:03.977275 27141 raft_consensus.cc:1273] T 7640f63d346f4d078ecb8d0e8a6c5670 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 3 FOLLOWER]: Refusing update from remote peer a5dbf236134d43158377e0310cee23d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:58:03.978323 27309 consensus_queue.cc:1035] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:58:03.985987 26992 raft_consensus.cc:1273] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [term 3 FOLLOWER]: Refusing update from remote peer a5dbf236134d43158377e0310cee23d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:58:03.988087 27321 consensus_queue.cc:1035] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:58:04.019776 26992 raft_consensus.cc:1273] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [term 3 FOLLOWER]: Refusing update from remote peer a5dbf236134d43158377e0310cee23d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:58:04.020766 27321 consensus_queue.cc:1035] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:58:04.028605 27141 raft_consensus.cc:1273] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 3 FOLLOWER]: Refusing update from remote peer a5dbf236134d43158377e0310cee23d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:58:04.029981 27325 consensus_queue.cc:1035] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:58:04.327942 26992 raft_consensus.cc:3058] T dbaec702dd3f48f2a1fcf12f8befed7e P ab704ad8154f4a4590e214cf9c1d7c4b [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:58:04.333269 26992 raft_consensus.cc:1273] T dbaec702dd3f48f2a1fcf12f8befed7e P ab704ad8154f4a4590e214cf9c1d7c4b [term 4 FOLLOWER]: Refusing update from remote peer a5dbf236134d43158377e0310cee23d7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250905 19:58:04.334393 27325 consensus_queue.cc:1035] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:58:04.348223 27141 raft_consensus.cc:1273] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 4 FOLLOWER]: Refusing update from remote peer a5dbf236134d43158377e0310cee23d7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250905 19:58:04.349725 27309 consensus_queue.cc:1035] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:58:06.421306 26858 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20250905 19:58:06.422410 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:06.429620 27066 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2f9344374d5649489c2f4336e9a1b15b"
dest_uuid: "a5dbf236134d43158377e0310cee23d7"
mode: GRACEFUL
new_leader_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
from {username='slave'} at 127.0.0.1:46142
I20250905 19:58:06.430295 27066 raft_consensus.cc:604] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [term 3 LEADER]: Received request to transfer leadership to TS ab704ad8154f4a4590e214cf9c1d7c4b
I20250905 19:58:06.432554 27066 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "71b161547e6a40e7b93b7f852b76e7d0"
dest_uuid: "a5dbf236134d43158377e0310cee23d7"
mode: GRACEFUL
new_leader_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
from {username='slave'} at 127.0.0.1:46142
I20250905 19:58:06.432999 27066 raft_consensus.cc:604] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [term 1 LEADER]: Received request to transfer leadership to TS ac86c30e5d044b9caf64b0c44f5cf7c8
I20250905 19:58:06.434911 27066 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7640f63d346f4d078ecb8d0e8a6c5670"
dest_uuid: "a5dbf236134d43158377e0310cee23d7"
mode: GRACEFUL
new_leader_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
from {username='slave'} at 127.0.0.1:46142
I20250905 19:58:06.435348 27066 raft_consensus.cc:604] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 3 LEADER]: Received request to transfer leadership to TS ab704ad8154f4a4590e214cf9c1d7c4b
I20250905 19:58:06.437209 27066 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "b2fabc1ff8894320aa02e55c92ecfb1e"
dest_uuid: "a5dbf236134d43158377e0310cee23d7"
mode: GRACEFUL
new_leader_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
from {username='slave'} at 127.0.0.1:46142
I20250905 19:58:06.437666 27066 raft_consensus.cc:604] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 [term 1 LEADER]: Received request to transfer leadership to TS ac86c30e5d044b9caf64b0c44f5cf7c8
I20250905 19:58:06.439603 27066 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c08349e8a8394ee182d5d0928074181f"
dest_uuid: "a5dbf236134d43158377e0310cee23d7"
mode: GRACEFUL
new_leader_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
from {username='slave'} at 127.0.0.1:46142
I20250905 19:58:06.440040 27066 raft_consensus.cc:604] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [term 3 LEADER]: Received request to transfer leadership to TS ab704ad8154f4a4590e214cf9c1d7c4b
I20250905 19:58:06.440907 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250905 19:58:06.441318 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:06.530079 27337 raft_consensus.cc:991] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7: : Instructing follower ac86c30e5d044b9caf64b0c44f5cf7c8 to start an election
I20250905 19:58:06.530619 27338 raft_consensus.cc:1079] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [term 1 LEADER]: Signalling peer ac86c30e5d044b9caf64b0c44f5cf7c8 to start an election
I20250905 19:58:06.531966 27142 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "71b161547e6a40e7b93b7f852b76e7d0"
dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
from {username='slave'} at 127.0.0.1:60404
I20250905 19:58:06.532363 27142 raft_consensus.cc:491] T 71b161547e6a40e7b93b7f852b76e7d0 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:58:06.532651 27142 raft_consensus.cc:3058] T 71b161547e6a40e7b93b7f852b76e7d0 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:06.537454 27142 raft_consensus.cc:513] T 71b161547e6a40e7b93b7f852b76e7d0 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:06.540506 27066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71b161547e6a40e7b93b7f852b76e7d0" candidate_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a5dbf236134d43158377e0310cee23d7"
I20250905 19:58:06.540458 26992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71b161547e6a40e7b93b7f852b76e7d0" candidate_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
I20250905 19:58:06.541296 27066 raft_consensus.cc:3053] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:58:06.539896 27142 leader_election.cc:290] T 71b161547e6a40e7b93b7f852b76e7d0 P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 2 election: Requested vote from peers ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107), a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875)
I20250905 19:58:06.541435 26992 raft_consensus.cc:3058] T 71b161547e6a40e7b93b7f852b76e7d0 P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:06.541631 27066 raft_consensus.cc:738] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a5dbf236134d43158377e0310cee23d7, State: Running, Role: LEADER
I20250905 19:58:06.542578 27066 consensus_queue.cc:260] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:06.543557 27066 raft_consensus.cc:3058] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:06.546144 26992 raft_consensus.cc:2466] T 71b161547e6a40e7b93b7f852b76e7d0 P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac86c30e5d044b9caf64b0c44f5cf7c8 in term 2.
I20250905 19:58:06.547237 27107 leader_election.cc:304] T 71b161547e6a40e7b93b7f852b76e7d0 P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab704ad8154f4a4590e214cf9c1d7c4b, ac86c30e5d044b9caf64b0c44f5cf7c8; no voters:
I20250905 19:58:06.548027 27313 raft_consensus.cc:2802] T 71b161547e6a40e7b93b7f852b76e7d0 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:58:06.548717 27066 raft_consensus.cc:2466] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac86c30e5d044b9caf64b0c44f5cf7c8 in term 2.
I20250905 19:58:06.549011 27313 raft_consensus.cc:695] T 71b161547e6a40e7b93b7f852b76e7d0 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 LEADER]: Becoming Leader. State: Replica: ac86c30e5d044b9caf64b0c44f5cf7c8, State: Running, Role: LEADER
I20250905 19:58:06.550000 27313 consensus_queue.cc:237] T 71b161547e6a40e7b93b7f852b76e7d0 P ac86c30e5d044b9caf64b0c44f5cf7c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:06.556444 26889 catalog_manager.cc:5582] T 71b161547e6a40e7b93b7f852b76e7d0 P ac86c30e5d044b9caf64b0c44f5cf7c8 reported cstate change: term changed from 1 to 2, leader changed from a5dbf236134d43158377e0310cee23d7 (127.26.58.130) to ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131). New cstate: current_term: 2 leader_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } health_report { overall_health: HEALTHY } } }
I20250905 19:58:06.565505 27337 raft_consensus.cc:991] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7: : Instructing follower ac86c30e5d044b9caf64b0c44f5cf7c8 to start an election
I20250905 19:58:06.566402 27338 raft_consensus.cc:1079] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 [term 1 LEADER]: Signalling peer ac86c30e5d044b9caf64b0c44f5cf7c8 to start an election
I20250905 19:58:06.568104 27142 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "b2fabc1ff8894320aa02e55c92ecfb1e"
dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
from {username='slave'} at 127.0.0.1:60404
I20250905 19:58:06.568666 27142 raft_consensus.cc:491] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:58:06.568991 27142 raft_consensus.cc:3058] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:06.573484 27142 raft_consensus.cc:513] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:06.574796 27142 leader_election.cc:290] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 2 election: Requested vote from peers ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107), a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875)
I20250905 19:58:06.575697 26992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2fabc1ff8894320aa02e55c92ecfb1e" candidate_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
I20250905 19:58:06.575681 27066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2fabc1ff8894320aa02e55c92ecfb1e" candidate_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a5dbf236134d43158377e0310cee23d7"
I20250905 19:58:06.576360 26992 raft_consensus.cc:3058] T b2fabc1ff8894320aa02e55c92ecfb1e P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:06.576431 27066 raft_consensus.cc:3053] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:58:06.576761 27066 raft_consensus.cc:738] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a5dbf236134d43158377e0310cee23d7, State: Running, Role: LEADER
I20250905 19:58:06.577389 27066 consensus_queue.cc:260] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:06.578301 27066 raft_consensus.cc:3058] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:06.580440 26992 raft_consensus.cc:2466] T b2fabc1ff8894320aa02e55c92ecfb1e P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac86c30e5d044b9caf64b0c44f5cf7c8 in term 2.
I20250905 19:58:06.581229 27107 leader_election.cc:304] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab704ad8154f4a4590e214cf9c1d7c4b, ac86c30e5d044b9caf64b0c44f5cf7c8; no voters:
I20250905 19:58:06.581945 27313 raft_consensus.cc:2802] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:58:06.582386 27313 raft_consensus.cc:695] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 LEADER]: Becoming Leader. State: Replica: ac86c30e5d044b9caf64b0c44f5cf7c8, State: Running, Role: LEADER
I20250905 19:58:06.582990 27066 raft_consensus.cc:2466] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac86c30e5d044b9caf64b0c44f5cf7c8 in term 2.
I20250905 19:58:06.583268 27313 consensus_queue.cc:237] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:06.591339 26889 catalog_manager.cc:5582] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8 reported cstate change: term changed from 1 to 2, leader changed from a5dbf236134d43158377e0310cee23d7 (127.26.58.130) to ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131). New cstate: current_term: 2 leader_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } health_report { overall_health: HEALTHY } } }
I20250905 19:58:06.647492 27345 raft_consensus.cc:991] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7: : Instructing follower ab704ad8154f4a4590e214cf9c1d7c4b to start an election
I20250905 19:58:06.647910 27338 raft_consensus.cc:1079] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 3 LEADER]: Signalling peer ab704ad8154f4a4590e214cf9c1d7c4b to start an election
I20250905 19:58:06.649140 26992 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7640f63d346f4d078ecb8d0e8a6c5670"
dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
from {username='slave'} at 127.0.0.1:48428
I20250905 19:58:06.649544 26992 raft_consensus.cc:491] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:58:06.649796 26992 raft_consensus.cc:3058] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:58:06.653669 26992 raft_consensus.cc:513] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:06.655624 26992 leader_election.cc:290] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 4 election: Requested vote from peers a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:58:06.656620 27066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7640f63d346f4d078ecb8d0e8a6c5670" candidate_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a5dbf236134d43158377e0310cee23d7"
I20250905 19:58:06.656726 27142 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7640f63d346f4d078ecb8d0e8a6c5670" candidate_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
I20250905 19:58:06.657320 27066 raft_consensus.cc:3053] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:58:06.657382 27142 raft_consensus.cc:3058] T 7640f63d346f4d078ecb8d0e8a6c5670 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:58:06.657716 27066 raft_consensus.cc:738] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a5dbf236134d43158377e0310cee23d7, State: Running, Role: LEADER
I20250905 19:58:06.658470 27066 consensus_queue.cc:260] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:06.659329 27066 raft_consensus.cc:3058] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:58:06.661845 27142 raft_consensus.cc:2466] T 7640f63d346f4d078ecb8d0e8a6c5670 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab704ad8154f4a4590e214cf9c1d7c4b in term 4.
I20250905 19:58:06.662925 26958 leader_election.cc:304] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab704ad8154f4a4590e214cf9c1d7c4b, ac86c30e5d044b9caf64b0c44f5cf7c8; no voters:
I20250905 19:58:06.663296 27066 raft_consensus.cc:2466] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab704ad8154f4a4590e214cf9c1d7c4b in term 4.
I20250905 19:58:06.664697 27347 raft_consensus.cc:2802] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:58:06.665629 27347 raft_consensus.cc:695] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [term 4 LEADER]: Becoming Leader. State: Replica: ab704ad8154f4a4590e214cf9c1d7c4b, State: Running, Role: LEADER
I20250905 19:58:06.666321 27347 consensus_queue.cc:237] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:06.673025 26889 catalog_manager.cc:5582] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b reported cstate change: term changed from 3 to 4, leader changed from a5dbf236134d43158377e0310cee23d7 (127.26.58.130) to ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129). New cstate: current_term: 4 leader_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:06.766005 27345 raft_consensus.cc:991] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7: : Instructing follower ab704ad8154f4a4590e214cf9c1d7c4b to start an election
I20250905 19:58:06.766398 27338 raft_consensus.cc:1079] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [term 3 LEADER]: Signalling peer ab704ad8154f4a4590e214cf9c1d7c4b to start an election
I20250905 19:58:06.767752 26992 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "2f9344374d5649489c2f4336e9a1b15b"
dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
from {username='slave'} at 127.0.0.1:48428
I20250905 19:58:06.768157 26992 raft_consensus.cc:491] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:58:06.768392 26992 raft_consensus.cc:3058] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:58:06.772500 26992 raft_consensus.cc:513] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:06.774083 26992 leader_election.cc:290] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 4 election: Requested vote from peers a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:58:06.774813 27066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f9344374d5649489c2f4336e9a1b15b" candidate_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a5dbf236134d43158377e0310cee23d7"
I20250905 19:58:06.775130 27142 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f9344374d5649489c2f4336e9a1b15b" candidate_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
I20250905 19:58:06.775381 27066 raft_consensus.cc:3053] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:58:06.775719 27066 raft_consensus.cc:738] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a5dbf236134d43158377e0310cee23d7, State: Running, Role: LEADER
I20250905 19:58:06.775693 27142 raft_consensus.cc:3058] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:58:06.776475 27066 consensus_queue.cc:260] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:06.777321 27066 raft_consensus.cc:3058] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:58:06.781196 27066 raft_consensus.cc:2466] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab704ad8154f4a4590e214cf9c1d7c4b in term 4.
I20250905 19:58:06.781903 27142 raft_consensus.cc:2466] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab704ad8154f4a4590e214cf9c1d7c4b in term 4.
I20250905 19:58:06.782137 26957 leader_election.cc:304] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5dbf236134d43158377e0310cee23d7, ab704ad8154f4a4590e214cf9c1d7c4b; no voters:
I20250905 19:58:06.782970 27347 raft_consensus.cc:2802] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:58:06.783416 27347 raft_consensus.cc:695] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [term 4 LEADER]: Becoming Leader. State: Replica: ab704ad8154f4a4590e214cf9c1d7c4b, State: Running, Role: LEADER
I20250905 19:58:06.784205 27347 consensus_queue.cc:237] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:06.791203 26889 catalog_manager.cc:5582] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b reported cstate change: term changed from 3 to 4, leader changed from a5dbf236134d43158377e0310cee23d7 (127.26.58.130) to ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129). New cstate: current_term: 4 leader_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:06.919533 27345 raft_consensus.cc:991] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7: : Instructing follower ab704ad8154f4a4590e214cf9c1d7c4b to start an election
I20250905 19:58:06.919965 27338 raft_consensus.cc:1079] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [term 3 LEADER]: Signalling peer ab704ad8154f4a4590e214cf9c1d7c4b to start an election
I20250905 19:58:06.921160 26992 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "c08349e8a8394ee182d5d0928074181f"
dest_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b"
from {username='slave'} at 127.0.0.1:48428
I20250905 19:58:06.921689 26992 raft_consensus.cc:491] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:58:06.922021 26992 raft_consensus.cc:3058] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:58:06.926666 26992 raft_consensus.cc:513] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:06.927927 26992 leader_election.cc:290] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 4 election: Requested vote from peers a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875), ac86c30e5d044b9caf64b0c44f5cf7c8 (127.26.58.131:35561)
I20250905 19:58:06.928776 27066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c08349e8a8394ee182d5d0928074181f" candidate_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a5dbf236134d43158377e0310cee23d7"
I20250905 19:58:06.928823 27142 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c08349e8a8394ee182d5d0928074181f" candidate_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8"
I20250905 19:58:06.929284 27066 raft_consensus.cc:3053] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:58:06.929324 27142 raft_consensus.cc:3058] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:58:06.929553 27066 raft_consensus.cc:738] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a5dbf236134d43158377e0310cee23d7, State: Running, Role: LEADER
I20250905 19:58:06.930130 27066 consensus_queue.cc:260] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:06.930999 27066 raft_consensus.cc:3058] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:58:06.933421 27142 raft_consensus.cc:2466] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab704ad8154f4a4590e214cf9c1d7c4b in term 4.
I20250905 19:58:06.934281 26958 leader_election.cc:304] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab704ad8154f4a4590e214cf9c1d7c4b, ac86c30e5d044b9caf64b0c44f5cf7c8; no voters:
I20250905 19:58:06.934778 27066 raft_consensus.cc:2466] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab704ad8154f4a4590e214cf9c1d7c4b in term 4.
I20250905 19:58:06.934978 27347 raft_consensus.cc:2802] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:58:06.935425 27347 raft_consensus.cc:695] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [term 4 LEADER]: Becoming Leader. State: Replica: ab704ad8154f4a4590e214cf9c1d7c4b, State: Running, Role: LEADER
I20250905 19:58:06.936220 27347 consensus_queue.cc:237] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } } peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } }
I20250905 19:58:06.942358 26889 catalog_manager.cc:5582] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b reported cstate change: term changed from 3 to 4, leader changed from a5dbf236134d43158377e0310cee23d7 (127.26.58.130) to ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129). New cstate: current_term: 4 leader_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:06.950148 27066 raft_consensus.cc:1273] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Refusing update from remote peer ac86c30e5d044b9caf64b0c44f5cf7c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:58:06.951267 27313 consensus_queue.cc:1035] T 71b161547e6a40e7b93b7f852b76e7d0 P ac86c30e5d044b9caf64b0c44f5cf7c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:58:06.963863 26992 raft_consensus.cc:1273] T 71b161547e6a40e7b93b7f852b76e7d0 P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 FOLLOWER]: Refusing update from remote peer ac86c30e5d044b9caf64b0c44f5cf7c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:58:06.965461 27354 consensus_queue.cc:1035] T 71b161547e6a40e7b93b7f852b76e7d0 P ac86c30e5d044b9caf64b0c44f5cf7c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:58:06.992035 26992 raft_consensus.cc:1273] T b2fabc1ff8894320aa02e55c92ecfb1e P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 FOLLOWER]: Refusing update from remote peer ac86c30e5d044b9caf64b0c44f5cf7c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:58:06.993095 27354 consensus_queue.cc:1035] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab704ad8154f4a4590e214cf9c1d7c4b" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 41107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:58:07.001317 27066 raft_consensus.cc:1273] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Refusing update from remote peer ac86c30e5d044b9caf64b0c44f5cf7c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:58:07.002493 27313 consensus_queue.cc:1035] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:58:07.061542 27066 raft_consensus.cc:1273] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 4 FOLLOWER]: Refusing update from remote peer ab704ad8154f4a4590e214cf9c1d7c4b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250905 19:58:07.062685 27347 consensus_queue.cc:1035] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:58:07.072274 27142 raft_consensus.cc:1273] T 7640f63d346f4d078ecb8d0e8a6c5670 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 4 FOLLOWER]: Refusing update from remote peer ab704ad8154f4a4590e214cf9c1d7c4b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250905 19:58:07.073777 27347 consensus_queue.cc:1035] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:58:07.322679 27066 raft_consensus.cc:1273] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [term 4 FOLLOWER]: Refusing update from remote peer ab704ad8154f4a4590e214cf9c1d7c4b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250905 19:58:07.323772 27347 consensus_queue.cc:1035] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:58:07.332192 27142 raft_consensus.cc:1273] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 4 FOLLOWER]: Refusing update from remote peer ab704ad8154f4a4590e214cf9c1d7c4b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250905 19:58:07.333438 27361 consensus_queue.cc:1035] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:58:07.344079 27142 raft_consensus.cc:1273] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 4 FOLLOWER]: Refusing update from remote peer ab704ad8154f4a4590e214cf9c1d7c4b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250905 19:58:07.345185 27361 consensus_queue.cc:1035] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac86c30e5d044b9caf64b0c44f5cf7c8" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 35561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:58:07.353394 27066 raft_consensus.cc:1273] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [term 4 FOLLOWER]: Refusing update from remote peer ab704ad8154f4a4590e214cf9c1d7c4b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250905 19:58:07.354576 27361 consensus_queue.cc:1035] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5dbf236134d43158377e0310cee23d7" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 44875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:58:07.442603 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:07.447752 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:07.448138 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:08.449105 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:08.454236 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:08.454870 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:09.456144 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:09.461458 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:09.461856 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:10.462925 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:10.467509 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:10.467839 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:11.468935 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:11.473307 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:11.473634 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:12.474610 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:12.479329 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:12.479660 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:13.480635 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:13.485249 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:13.485648 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:14.486732 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:14.491755 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:14.492142 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:15.493064 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:15.499492 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:15.499902 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:16.526000 26858 auto_leader_rebalancer-test.cc:318] The leader distribution is
I20250905 19:58:16.526932 26858 tablet_server.cc:178] TabletServer@127.26.58.129:0 shutting down...
I20250905 19:58:16.552234 26858 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:58:16.552915 26858 tablet_replica.cc:331] T 71b161547e6a40e7b93b7f852b76e7d0 P ab704ad8154f4a4590e214cf9c1d7c4b: stopping tablet replica
I20250905 19:58:16.553843 26858 raft_consensus.cc:2241] T 71b161547e6a40e7b93b7f852b76e7d0 P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:16.554544 26858 raft_consensus.cc:2270] T 71b161547e6a40e7b93b7f852b76e7d0 P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:16.557010 26858 tablet_replica.cc:331] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b: stopping tablet replica
I20250905 19:58:16.557590 26858 raft_consensus.cc:2241] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [term 4 LEADER]: Raft consensus shutting down.
I20250905 19:58:16.558358 26858 raft_consensus.cc:2270] T 2f9344374d5649489c2f4336e9a1b15b P ab704ad8154f4a4590e214cf9c1d7c4b [term 4 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:16.560433 26858 tablet_replica.cc:331] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b: stopping tablet replica
I20250905 19:58:16.561053 26858 raft_consensus.cc:2241] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [term 4 LEADER]: Raft consensus shutting down.
W20250905 19:58:16.561971 27107 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250905 19:58:16.562086 26858 raft_consensus.cc:2270] T c08349e8a8394ee182d5d0928074181f P ab704ad8154f4a4590e214cf9c1d7c4b [term 4 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:16.564340 26858 tablet_replica.cc:331] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b: stopping tablet replica
I20250905 19:58:16.564901 26858 raft_consensus.cc:2241] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:16.565467 26858 raft_consensus.cc:2270] T 685a60cc8a6841c6808fcba893636187 P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 FOLLOWER]: Raft consensus is shut down!
W20250905 19:58:16.565483 27107 consensus_peers.cc:489] T 71b161547e6a40e7b93b7f852b76e7d0 P ac86c30e5d044b9caf64b0c44f5cf7c8 -> Peer ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107): Couldn't send request to peer ab704ad8154f4a4590e214cf9c1d7c4b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:58:16.567250 26858 tablet_replica.cc:331] T f6cfad67eeab45ddb61ef84d09f09b6b P ab704ad8154f4a4590e214cf9c1d7c4b: stopping tablet replica
I20250905 19:58:16.567761 26858 raft_consensus.cc:2241] T f6cfad67eeab45ddb61ef84d09f09b6b P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:16.568128 26858 raft_consensus.cc:2270] T f6cfad67eeab45ddb61ef84d09f09b6b P ab704ad8154f4a4590e214cf9c1d7c4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:16.569808 26858 tablet_replica.cc:331] T b2fabc1ff8894320aa02e55c92ecfb1e P ab704ad8154f4a4590e214cf9c1d7c4b: stopping tablet replica
I20250905 19:58:16.570289 26858 raft_consensus.cc:2241] T b2fabc1ff8894320aa02e55c92ecfb1e P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:16.570755 26858 raft_consensus.cc:2270] T b2fabc1ff8894320aa02e55c92ecfb1e P ab704ad8154f4a4590e214cf9c1d7c4b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:16.572278 26858 tablet_replica.cc:331] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b: stopping tablet replica
I20250905 19:58:16.572721 26858 raft_consensus.cc:2241] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b [term 3 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:16.573127 26858 raft_consensus.cc:2270] T c6a00cc443b94edcb46fbfdf1d7ac32c P ab704ad8154f4a4590e214cf9c1d7c4b [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:16.574749 26858 tablet_replica.cc:331] T dbaec702dd3f48f2a1fcf12f8befed7e P ab704ad8154f4a4590e214cf9c1d7c4b: stopping tablet replica
I20250905 19:58:16.575197 26858 raft_consensus.cc:2241] T dbaec702dd3f48f2a1fcf12f8befed7e P ab704ad8154f4a4590e214cf9c1d7c4b [term 4 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:16.575603 26858 raft_consensus.cc:2270] T dbaec702dd3f48f2a1fcf12f8befed7e P ab704ad8154f4a4590e214cf9c1d7c4b [term 4 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:16.577167 26858 tablet_replica.cc:331] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b: stopping tablet replica
I20250905 19:58:16.577589 26858 raft_consensus.cc:2241] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [term 4 LEADER]: Raft consensus shutting down.
I20250905 19:58:16.578365 26858 raft_consensus.cc:2270] T 7640f63d346f4d078ecb8d0e8a6c5670 P ab704ad8154f4a4590e214cf9c1d7c4b [term 4 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:16.695331 26858 tablet_server.cc:195] TabletServer@127.26.58.129:0 shutdown complete.
W20250905 19:58:16.695922 27033 consensus_peers.cc:489] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 -> Peer ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107): Couldn't send request to peer ab704ad8154f4a4590e214cf9c1d7c4b. Status: Network error: Client connection negotiation failed: client connection to 127.26.58.129:41107: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 19:58:16.713996 26858 tablet_server.cc:178] TabletServer@127.26.58.130:0 shutting down...
W20250905 19:58:16.729823 27107 consensus_peers.cc:489] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8 -> Peer ab704ad8154f4a4590e214cf9c1d7c4b (127.26.58.129:41107): Couldn't send request to peer ab704ad8154f4a4590e214cf9c1d7c4b. Status: Network error: Client connection negotiation failed: client connection to 127.26.58.129:41107: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 19:58:16.741990 26858 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:58:16.742738 26858 tablet_replica.cc:331] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7: stopping tablet replica
I20250905 19:58:16.743830 26858 raft_consensus.cc:2241] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:16.744441 26858 raft_consensus.cc:2270] T c08349e8a8394ee182d5d0928074181f P a5dbf236134d43158377e0310cee23d7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:16.746165 26858 tablet_replica.cc:331] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7: stopping tablet replica
I20250905 19:58:16.746683 26858 raft_consensus.cc:2241] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:16.747042 26858 raft_consensus.cc:2270] T 2f9344374d5649489c2f4336e9a1b15b P a5dbf236134d43158377e0310cee23d7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:16.748538 26858 tablet_replica.cc:331] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7: stopping tablet replica
I20250905 19:58:16.748965 26858 raft_consensus.cc:2241] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:16.749312 26858 raft_consensus.cc:2270] T b2fabc1ff8894320aa02e55c92ecfb1e P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:16.750921 26858 tablet_replica.cc:331] T c6a00cc443b94edcb46fbfdf1d7ac32c P a5dbf236134d43158377e0310cee23d7: stopping tablet replica
I20250905 19:58:16.751361 26858 raft_consensus.cc:2241] T c6a00cc443b94edcb46fbfdf1d7ac32c P a5dbf236134d43158377e0310cee23d7 [term 3 LEADER]: Raft consensus shutting down.
I20250905 19:58:16.752112 26858 raft_consensus.cc:2270] T c6a00cc443b94edcb46fbfdf1d7ac32c P a5dbf236134d43158377e0310cee23d7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:16.753592 26858 tablet_replica.cc:331] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7: stopping tablet replica
I20250905 19:58:16.754010 26858 raft_consensus.cc:2241] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 4 LEADER]: Raft consensus shutting down.
I20250905 19:58:16.754832 26858 raft_consensus.cc:2270] T dbaec702dd3f48f2a1fcf12f8befed7e P a5dbf236134d43158377e0310cee23d7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:16.756522 26858 tablet_replica.cc:331] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7: stopping tablet replica
I20250905 19:58:16.756979 26858 raft_consensus.cc:2241] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:16.757359 26858 raft_consensus.cc:2270] T 7640f63d346f4d078ecb8d0e8a6c5670 P a5dbf236134d43158377e0310cee23d7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:16.758898 26858 tablet_replica.cc:331] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7: stopping tablet replica
I20250905 19:58:16.759346 26858 raft_consensus.cc:2241] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:16.759748 26858 raft_consensus.cc:2270] T 71b161547e6a40e7b93b7f852b76e7d0 P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:16.761319 26858 tablet_replica.cc:331] T f6cfad67eeab45ddb61ef84d09f09b6b P a5dbf236134d43158377e0310cee23d7: stopping tablet replica
I20250905 19:58:16.761771 26858 raft_consensus.cc:2241] T f6cfad67eeab45ddb61ef84d09f09b6b P a5dbf236134d43158377e0310cee23d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:16.762141 26858 raft_consensus.cc:2270] T f6cfad67eeab45ddb61ef84d09f09b6b P a5dbf236134d43158377e0310cee23d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:16.763697 26858 tablet_replica.cc:331] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7: stopping tablet replica
I20250905 19:58:16.764088 26858 raft_consensus.cc:2241] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7 [term 2 LEADER]: Raft consensus shutting down.
I20250905 19:58:16.764850 26858 raft_consensus.cc:2270] T 685a60cc8a6841c6808fcba893636187 P a5dbf236134d43158377e0310cee23d7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:16.791209 26858 tablet_server.cc:195] TabletServer@127.26.58.130:0 shutdown complete.
I20250905 19:58:16.808851 26858 tablet_server.cc:178] TabletServer@127.26.58.131:0 shutting down...
W20250905 19:58:16.815161 27106 consensus_peers.cc:489] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8 -> Peer a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875): Couldn't send request to peer a5dbf236134d43158377e0310cee23d7. Status: Network error: Client connection negotiation failed: client connection to 127.26.58.130:44875: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250905 19:58:16.841096 27106 consensus_peers.cc:489] T 71b161547e6a40e7b93b7f852b76e7d0 P ac86c30e5d044b9caf64b0c44f5cf7c8 -> Peer a5dbf236134d43158377e0310cee23d7 (127.26.58.130:44875): Couldn't send request to peer a5dbf236134d43158377e0310cee23d7. Status: Network error: Client connection negotiation failed: client connection to 127.26.58.130:44875: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 19:58:16.842307 26858 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:58:16.842928 26858 tablet_replica.cc:331] T 7640f63d346f4d078ecb8d0e8a6c5670 P ac86c30e5d044b9caf64b0c44f5cf7c8: stopping tablet replica
I20250905 19:58:16.843499 26858 raft_consensus.cc:2241] T 7640f63d346f4d078ecb8d0e8a6c5670 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:16.843921 26858 raft_consensus.cc:2270] T 7640f63d346f4d078ecb8d0e8a6c5670 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:16.845587 26858 tablet_replica.cc:331] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8: stopping tablet replica
I20250905 19:58:16.846096 26858 raft_consensus.cc:2241] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:16.846546 26858 raft_consensus.cc:2270] T dbaec702dd3f48f2a1fcf12f8befed7e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:16.848168 26858 tablet_replica.cc:331] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8: stopping tablet replica
I20250905 19:58:16.848649 26858 raft_consensus.cc:2241] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:16.849071 26858 raft_consensus.cc:2270] T c6a00cc443b94edcb46fbfdf1d7ac32c P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:16.850641 26858 tablet_replica.cc:331] T 71b161547e6a40e7b93b7f852b76e7d0 P ac86c30e5d044b9caf64b0c44f5cf7c8: stopping tablet replica
I20250905 19:58:16.851112 26858 raft_consensus.cc:2241] T 71b161547e6a40e7b93b7f852b76e7d0 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 LEADER]: Raft consensus shutting down.
I20250905 19:58:16.851853 26858 raft_consensus.cc:2270] T 71b161547e6a40e7b93b7f852b76e7d0 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:16.853490 26858 tablet_replica.cc:331] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8: stopping tablet replica
I20250905 19:58:16.853945 26858 raft_consensus.cc:2241] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:16.854329 26858 raft_consensus.cc:2270] T c08349e8a8394ee182d5d0928074181f P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:16.855922 26858 tablet_replica.cc:331] T 685a60cc8a6841c6808fcba893636187 P ac86c30e5d044b9caf64b0c44f5cf7c8: stopping tablet replica
I20250905 19:58:16.856375 26858 raft_consensus.cc:2241] T 685a60cc8a6841c6808fcba893636187 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:16.856772 26858 raft_consensus.cc:2270] T 685a60cc8a6841c6808fcba893636187 P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:16.858233 26858 tablet_replica.cc:331] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8: stopping tablet replica
I20250905 19:58:16.858726 26858 raft_consensus.cc:2241] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:58:16.859414 26858 raft_consensus.cc:2270] T f6cfad67eeab45ddb61ef84d09f09b6b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:16.860996 26858 tablet_replica.cc:331] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8: stopping tablet replica
I20250905 19:58:16.861430 26858 raft_consensus.cc:2241] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:16.861857 26858 raft_consensus.cc:2270] T 2f9344374d5649489c2f4336e9a1b15b P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:16.863588 26858 tablet_replica.cc:331] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8: stopping tablet replica
I20250905 19:58:16.864012 26858 raft_consensus.cc:2241] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 LEADER]: Raft consensus shutting down.
I20250905 19:58:16.864804 26858 raft_consensus.cc:2270] T b2fabc1ff8894320aa02e55c92ecfb1e P ac86c30e5d044b9caf64b0c44f5cf7c8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:16.912376 26858 tablet_server.cc:195] TabletServer@127.26.58.131:0 shutdown complete.
I20250905 19:58:16.930246 26858 master.cc:561] Master@127.26.58.190:39653 shutting down...
I20250905 19:58:17.035089 26858 raft_consensus.cc:2241] T 00000000000000000000000000000000 P fe6db7b21b684612a333a0c0d71626ef [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:58:17.035712 26858 raft_consensus.cc:2270] T 00000000000000000000000000000000 P fe6db7b21b684612a333a0c0d71626ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:17.035996 26858 tablet_replica.cc:331] T 00000000000000000000000000000000 P fe6db7b21b684612a333a0c0d71626ef: stopping tablet replica
I20250905 19:58:17.055253 26858 master.cc:583] Master@127.26.58.190:39653 shutdown complete.
a5dbf236134d43158377e0310cee23d7 4
ab704ad8154f4a4590e214cf9c1d7c4b 4
ac86c30e5d044b9caf64b0c44f5cf7c8 1
a5dbf236134d43158377e0310cee23d7 3
ab704ad8154f4a4590e214cf9c1d7c4b 3
ac86c30e5d044b9caf64b0c44f5cf7c8 3
a5dbf236134d43158377e0310cee23d7 8
ab704ad8154f4a4590e214cf9c1d7c4b 0
ac86c30e5d044b9caf64b0c44f5cf7c8 1
a5dbf236134d43158377e0310cee23d7 3
ab704ad8154f4a4590e214cf9c1d7c4b 3
ac86c30e5d044b9caf64b0c44f5cf7c8 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (31180 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20250905 19:58:17.093696 26858 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.26.58.190:43327
I20250905 19:58:17.094821 26858 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:58:17.100201 27379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:58:17.100493 27380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:58:17.103147 26858 server_base.cc:1047] running on GCE node
W20250905 19:58:17.103109 27382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:58:17.104297 26858 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:58:17.104488 26858 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:58:17.104660 26858 hybrid_clock.cc:648] HybridClock initialized: now 1757102297104638 us; error 0 us; skew 500 ppm
I20250905 19:58:17.105159 26858 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:58:17.107679 26858 webserver.cc:480] Webserver started at http://127.26.58.190:38767/ using document root <none> and password file <none>
I20250905 19:58:17.108153 26858 fs_manager.cc:362] Metadata directory not provided
I20250905 19:58:17.108332 26858 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:58:17.108585 26858 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:58:17.109694 26858 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1757102265724956-26858-0/minicluster-data/master-0-root/instance:
uuid: "14cfe5353e384054adfb13763c0386d8"
format_stamp: "Formatted at 2025-09-05 19:58:17 on dist-test-slave-v26w"
I20250905 19:58:17.113921 26858 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250905 19:58:17.117059 27387 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:58:17.117740 26858 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250905 19:58:17.117997 26858 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1757102265724956-26858-0/minicluster-data/master-0-root
uuid: "14cfe5353e384054adfb13763c0386d8"
format_stamp: "Formatted at 2025-09-05 19:58:17 on dist-test-slave-v26w"
I20250905 19:58:17.118254 26858 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1757102265724956-26858-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1757102265724956-26858-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1757102265724956-26858-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:58:17.131374 26858 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:58:17.132563 26858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:58:17.169214 26858 rpc_server.cc:307] RPC server started. Bound to: 127.26.58.190:43327
I20250905 19:58:17.169294 27438 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.58.190:43327 every 8 connection(s)
I20250905 19:58:17.173007 27439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.182533 27439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14cfe5353e384054adfb13763c0386d8: Bootstrap starting.
I20250905 19:58:17.187124 27439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14cfe5353e384054adfb13763c0386d8: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:17.191213 27439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14cfe5353e384054adfb13763c0386d8: No bootstrap required, opened a new log
I20250905 19:58:17.193272 27439 raft_consensus.cc:357] T 00000000000000000000000000000000 P 14cfe5353e384054adfb13763c0386d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cfe5353e384054adfb13763c0386d8" member_type: VOTER }
I20250905 19:58:17.193713 27439 raft_consensus.cc:383] T 00000000000000000000000000000000 P 14cfe5353e384054adfb13763c0386d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:17.193964 27439 raft_consensus.cc:738] T 00000000000000000000000000000000 P 14cfe5353e384054adfb13763c0386d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14cfe5353e384054adfb13763c0386d8, State: Initialized, Role: FOLLOWER
I20250905 19:58:17.194541 27439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14cfe5353e384054adfb13763c0386d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cfe5353e384054adfb13763c0386d8" member_type: VOTER }
I20250905 19:58:17.194988 27439 raft_consensus.cc:397] T 00000000000000000000000000000000 P 14cfe5353e384054adfb13763c0386d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 19:58:17.195227 27439 raft_consensus.cc:491] T 00000000000000000000000000000000 P 14cfe5353e384054adfb13763c0386d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 19:58:17.195500 27439 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 14cfe5353e384054adfb13763c0386d8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:17.199412 27439 raft_consensus.cc:513] T 00000000000000000000000000000000 P 14cfe5353e384054adfb13763c0386d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cfe5353e384054adfb13763c0386d8" member_type: VOTER }
I20250905 19:58:17.199888 27439 leader_election.cc:304] T 00000000000000000000000000000000 P 14cfe5353e384054adfb13763c0386d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14cfe5353e384054adfb13763c0386d8; no voters:
I20250905 19:58:17.200934 27439 leader_election.cc:290] T 00000000000000000000000000000000 P 14cfe5353e384054adfb13763c0386d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 19:58:17.201309 27442 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 14cfe5353e384054adfb13763c0386d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:17.202654 27442 raft_consensus.cc:695] T 00000000000000000000000000000000 P 14cfe5353e384054adfb13763c0386d8 [term 1 LEADER]: Becoming Leader. State: Replica: 14cfe5353e384054adfb13763c0386d8, State: Running, Role: LEADER
I20250905 19:58:17.203223 27442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14cfe5353e384054adfb13763c0386d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cfe5353e384054adfb13763c0386d8" member_type: VOTER }
I20250905 19:58:17.203891 27439 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14cfe5353e384054adfb13763c0386d8 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 19:58:17.208192 27443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14cfe5353e384054adfb13763c0386d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14cfe5353e384054adfb13763c0386d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cfe5353e384054adfb13763c0386d8" member_type: VOTER } }
I20250905 19:58:17.208966 27444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14cfe5353e384054adfb13763c0386d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14cfe5353e384054adfb13763c0386d8. Latest consensus state: current_term: 1 leader_uuid: "14cfe5353e384054adfb13763c0386d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cfe5353e384054adfb13763c0386d8" member_type: VOTER } }
I20250905 19:58:17.209055 27443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14cfe5353e384054adfb13763c0386d8 [sys.catalog]: This master's current role is: LEADER
I20250905 19:58:17.209450 27444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14cfe5353e384054adfb13763c0386d8 [sys.catalog]: This master's current role is: LEADER
I20250905 19:58:17.211201 27449 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 19:58:17.216652 27449 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 19:58:17.219700 26858 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250905 19:58:17.225560 27449 catalog_manager.cc:1349] Generated new cluster ID: 5805e2f255c64ee08c8171c92a2b3c01
I20250905 19:58:17.225832 27449 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 19:58:17.257906 27449 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 19:58:17.259133 27449 catalog_manager.cc:1506] Loading token signing keys...
I20250905 19:58:17.271657 27449 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 14cfe5353e384054adfb13763c0386d8: Generated new TSK 0
I20250905 19:58:17.272223 27449 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 19:58:17.286695 26858 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:58:17.292346 27460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:58:17.293535 27461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:58:17.295853 27463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:58:17.295905 26858 server_base.cc:1047] running on GCE node
I20250905 19:58:17.297009 26858 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:58:17.297223 26858 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:58:17.297387 26858 hybrid_clock.cc:648] HybridClock initialized: now 1757102297297369 us; error 0 us; skew 500 ppm
I20250905 19:58:17.297897 26858 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:58:17.300204 26858 webserver.cc:480] Webserver started at http://127.26.58.129:37173/ using document root <none> and password file <none>
I20250905 19:58:17.300686 26858 fs_manager.cc:362] Metadata directory not provided
I20250905 19:58:17.300870 26858 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:58:17.301117 26858 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:58:17.302219 26858 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1757102265724956-26858-0/minicluster-data/ts-0-root/instance:
uuid: "025319773a4247b99829aa83dd002b0f"
format_stamp: "Formatted at 2025-09-05 19:58:17 on dist-test-slave-v26w"
I20250905 19:58:17.306471 26858 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250905 19:58:17.309471 27468 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:58:17.310159 26858 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20250905 19:58:17.310463 26858 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1757102265724956-26858-0/minicluster-data/ts-0-root
uuid: "025319773a4247b99829aa83dd002b0f"
format_stamp: "Formatted at 2025-09-05 19:58:17 on dist-test-slave-v26w"
I20250905 19:58:17.310865 26858 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1757102265724956-26858-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1757102265724956-26858-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1757102265724956-26858-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:58:17.325278 26858 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:58:17.326611 26858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:58:17.328071 26858 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:58:17.330327 26858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:58:17.330621 26858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:58:17.330878 26858 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:58:17.331051 26858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:58:17.371555 26858 rpc_server.cc:307] RPC server started. Bound to: 127.26.58.129:43725
I20250905 19:58:17.371649 27530 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.58.129:43725 every 8 connection(s)
I20250905 19:58:17.376259 26858 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:58:17.383885 27535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:58:17.384107 27536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:58:17.387748 26858 server_base.cc:1047] running on GCE node
W20250905 19:58:17.388813 27538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:58:17.389773 26858 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250905 19:58:17.389823 27531 heartbeater.cc:344] Connected to a master server at 127.26.58.190:43327
W20250905 19:58:17.390136 26858 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:58:17.390321 27531 heartbeater.cc:461] Registering TS with master...
I20250905 19:58:17.390352 26858 hybrid_clock.cc:648] HybridClock initialized: now 1757102297390334 us; error 0 us; skew 500 ppm
I20250905 19:58:17.391196 26858 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:58:17.391314 27531 heartbeater.cc:507] Master 127.26.58.190:43327 requested a full tablet report, sending...
I20250905 19:58:17.393537 27404 ts_manager.cc:194] Registered new tserver with Master: 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725)
I20250905 19:58:17.393998 26858 webserver.cc:480] Webserver started at http://127.26.58.130:46433/ using document root <none> and password file <none>
I20250905 19:58:17.394533 26858 fs_manager.cc:362] Metadata directory not provided
I20250905 19:58:17.394717 26858 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:58:17.394960 26858 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:58:17.395462 27404 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39694
I20250905 19:58:17.396194 26858 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1757102265724956-26858-0/minicluster-data/ts-1-root/instance:
uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
format_stamp: "Formatted at 2025-09-05 19:58:17 on dist-test-slave-v26w"
I20250905 19:58:17.400900 26858 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250905 19:58:17.403980 27543 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:58:17.404735 26858 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250905 19:58:17.405016 26858 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1757102265724956-26858-0/minicluster-data/ts-1-root
uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
format_stamp: "Formatted at 2025-09-05 19:58:17 on dist-test-slave-v26w"
I20250905 19:58:17.405342 26858 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1757102265724956-26858-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1757102265724956-26858-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1757102265724956-26858-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:58:17.423029 26858 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:58:17.424198 26858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:58:17.425648 26858 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:58:17.427783 26858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:58:17.427978 26858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:58:17.428198 26858 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:58:17.428357 26858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:58:17.466871 26858 rpc_server.cc:307] RPC server started. Bound to: 127.26.58.130:43929
I20250905 19:58:17.466964 27605 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.58.130:43929 every 8 connection(s)
I20250905 19:58:17.471382 26858 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:58:17.478549 27609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:58:17.480579 27610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:58:17.482208 26858 server_base.cc:1047] running on GCE node
W20250905 19:58:17.482771 27612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:58:17.483763 26858 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:58:17.484030 26858 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:58:17.484205 27606 heartbeater.cc:344] Connected to a master server at 127.26.58.190:43327
I20250905 19:58:17.484248 26858 hybrid_clock.cc:648] HybridClock initialized: now 1757102297484228 us; error 0 us; skew 500 ppm
I20250905 19:58:17.484658 27606 heartbeater.cc:461] Registering TS with master...
I20250905 19:58:17.485019 26858 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:58:17.485389 27606 heartbeater.cc:507] Master 127.26.58.190:43327 requested a full tablet report, sending...
I20250905 19:58:17.487679 26858 webserver.cc:480] Webserver started at http://127.26.58.131:42195/ using document root <none> and password file <none>
I20250905 19:58:17.487653 27404 ts_manager.cc:194] Registered new tserver with Master: 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929)
I20250905 19:58:17.488402 26858 fs_manager.cc:362] Metadata directory not provided
I20250905 19:58:17.488608 26858 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:58:17.488842 26858 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:58:17.489307 27404 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39706
I20250905 19:58:17.489967 26858 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1757102265724956-26858-0/minicluster-data/ts-2-root/instance:
uuid: "15a3df6778944e9c8d1b663004a83903"
format_stamp: "Formatted at 2025-09-05 19:58:17 on dist-test-slave-v26w"
I20250905 19:58:17.494740 26858 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20250905 19:58:17.497807 27617 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:58:17.498544 26858 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250905 19:58:17.498804 26858 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1757102265724956-26858-0/minicluster-data/ts-2-root
uuid: "15a3df6778944e9c8d1b663004a83903"
format_stamp: "Formatted at 2025-09-05 19:58:17 on dist-test-slave-v26w"
I20250905 19:58:17.499066 26858 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1757102265724956-26858-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1757102265724956-26858-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1757102265724956-26858-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:58:17.516389 26858 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:58:17.517505 26858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:58:17.518929 26858 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:58:17.521112 26858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:58:17.521307 26858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:58:17.521538 26858 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:58:17.521694 26858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:58:17.560580 26858 rpc_server.cc:307] RPC server started. Bound to: 127.26.58.131:42617
I20250905 19:58:17.560648 27679 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.58.131:42617 every 8 connection(s)
I20250905 19:58:17.573594 27680 heartbeater.cc:344] Connected to a master server at 127.26.58.190:43327
I20250905 19:58:17.573927 27680 heartbeater.cc:461] Registering TS with master...
I20250905 19:58:17.574626 27680 heartbeater.cc:507] Master 127.26.58.190:43327 requested a full tablet report, sending...
I20250905 19:58:17.576346 27404 ts_manager.cc:194] Registered new tserver with Master: 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:17.576742 26858 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013178333s
I20250905 19:58:17.578001 27404 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39712
I20250905 19:58:17.579564 26858 test_util.cc:276] Using random seed: 1112014864
I20250905 19:58:17.601302 27404 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39726:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:58:17.604059 27404 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:58:17.683435 27571 tablet_service.cc:1468] Processing CreateTablet for tablet 1a6727b25590409b9402c5aee52a55a7 (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250905 19:58:17.684934 27571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a6727b25590409b9402c5aee52a55a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.685830 27564 tablet_service.cc:1468] Processing CreateTablet for tablet 3ad0f154c598458fa48d8fd40ce46171 (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250905 19:58:17.687242 27564 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ad0f154c598458fa48d8fd40ce46171. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.687644 27570 tablet_service.cc:1468] Processing CreateTablet for tablet fa10aea786b5462c82a92fc0f9bff996 (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250905 19:58:17.688966 27570 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa10aea786b5462c82a92fc0f9bff996. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.688930 27563 tablet_service.cc:1468] Processing CreateTablet for tablet 650634b3e04147809fdde8cf72063c78 (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250905 19:58:17.692103 27569 tablet_service.cc:1468] Processing CreateTablet for tablet 66b1b69c203145e4bcc2cb26522c663f (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250905 19:58:17.692399 27491 tablet_service.cc:1468] Processing CreateTablet for tablet 7fa1390a6ad647498e7b9459277a60ac (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250905 19:58:17.693724 27491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fa1390a6ad647498e7b9459277a60ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.692251 27492 tablet_service.cc:1468] Processing CreateTablet for tablet 25d5dc671025465b90a24a0f20dae08f (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250905 19:58:17.694361 27566 tablet_service.cc:1468] Processing CreateTablet for tablet 7fa1390a6ad647498e7b9459277a60ac (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250905 19:58:17.698077 27562 tablet_service.cc:1468] Processing CreateTablet for tablet 02a7a3e0f7a74cc2a1d2762c05eca777 (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250905 19:58:17.697103 27567 tablet_service.cc:1468] Processing CreateTablet for tablet 25d5dc671025465b90a24a0f20dae08f (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250905 19:58:17.698886 27492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25d5dc671025465b90a24a0f20dae08f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.696071 27568 tablet_service.cc:1468] Processing CreateTablet for tablet 5eb30fb918af423aba649dba5b74f749 (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250905 19:58:17.695747 27494 tablet_service.cc:1468] Processing CreateTablet for tablet 66b1b69c203145e4bcc2cb26522c663f (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250905 19:58:17.700237 27495 tablet_service.cc:1468] Processing CreateTablet for tablet fa10aea786b5462c82a92fc0f9bff996 (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250905 19:58:17.692106 27565 tablet_service.cc:1468] Processing CreateTablet for tablet ccec5ef5fb0d4383a93e6cf0514aefb8 (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250905 19:58:17.701350 27494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66b1b69c203145e4bcc2cb26522c663f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.702343 27490 tablet_service.cc:1468] Processing CreateTablet for tablet ccec5ef5fb0d4383a93e6cf0514aefb8 (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250905 19:58:17.704200 27487 tablet_service.cc:1468] Processing CreateTablet for tablet 02a7a3e0f7a74cc2a1d2762c05eca777 (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250905 19:58:17.704732 27496 tablet_service.cc:1468] Processing CreateTablet for tablet 1a6727b25590409b9402c5aee52a55a7 (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250905 19:58:17.695510 27493 tablet_service.cc:1468] Processing CreateTablet for tablet 5eb30fb918af423aba649dba5b74f749 (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250905 19:58:17.690290 27563 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 650634b3e04147809fdde8cf72063c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.707525 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5eb30fb918af423aba649dba5b74f749. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.707823 27562 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02a7a3e0f7a74cc2a1d2762c05eca777. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.696900 27489 tablet_service.cc:1468] Processing CreateTablet for tablet 3ad0f154c598458fa48d8fd40ce46171 (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250905 19:58:17.708467 27487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02a7a3e0f7a74cc2a1d2762c05eca777. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.694087 27488 tablet_service.cc:1468] Processing CreateTablet for tablet 650634b3e04147809fdde8cf72063c78 (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250905 19:58:17.708746 27566 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fa1390a6ad647498e7b9459277a60ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.709142 27490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccec5ef5fb0d4383a93e6cf0514aefb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.710219 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a6727b25590409b9402c5aee52a55a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.710328 27568 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5eb30fb918af423aba649dba5b74f749. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.712417 27567 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25d5dc671025465b90a24a0f20dae08f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.720878 27569 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66b1b69c203145e4bcc2cb26522c663f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.721606 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa10aea786b5462c82a92fc0f9bff996. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.722164 27565 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccec5ef5fb0d4383a93e6cf0514aefb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.723726 27489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ad0f154c598458fa48d8fd40ce46171. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.727635 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 650634b3e04147809fdde8cf72063c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.755620 27645 tablet_service.cc:1468] Processing CreateTablet for tablet 1a6727b25590409b9402c5aee52a55a7 (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250905 19:58:17.756995 27645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a6727b25590409b9402c5aee52a55a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.759537 27644 tablet_service.cc:1468] Processing CreateTablet for tablet fa10aea786b5462c82a92fc0f9bff996 (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250905 19:58:17.760965 27644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa10aea786b5462c82a92fc0f9bff996. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.763610 27643 tablet_service.cc:1468] Processing CreateTablet for tablet 66b1b69c203145e4bcc2cb26522c663f (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250905 19:58:17.764976 27643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66b1b69c203145e4bcc2cb26522c663f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.767630 27642 tablet_service.cc:1468] Processing CreateTablet for tablet 5eb30fb918af423aba649dba5b74f749 (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250905 19:58:17.768960 27642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5eb30fb918af423aba649dba5b74f749. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.771641 27641 tablet_service.cc:1468] Processing CreateTablet for tablet 25d5dc671025465b90a24a0f20dae08f (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250905 19:58:17.772974 27641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25d5dc671025465b90a24a0f20dae08f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.777031 27640 tablet_service.cc:1468] Processing CreateTablet for tablet 7fa1390a6ad647498e7b9459277a60ac (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250905 19:58:17.778349 27640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fa1390a6ad647498e7b9459277a60ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.778195 27639 tablet_service.cc:1468] Processing CreateTablet for tablet ccec5ef5fb0d4383a93e6cf0514aefb8 (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250905 19:58:17.779436 27639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccec5ef5fb0d4383a93e6cf0514aefb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.785023 27638 tablet_service.cc:1468] Processing CreateTablet for tablet 3ad0f154c598458fa48d8fd40ce46171 (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250905 19:58:17.789808 27637 tablet_service.cc:1468] Processing CreateTablet for tablet 650634b3e04147809fdde8cf72063c78 (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250905 19:58:17.793370 27636 tablet_service.cc:1468] Processing CreateTablet for tablet 02a7a3e0f7a74cc2a1d2762c05eca777 (DEFAULT_TABLE table=test-workload [id=796b6f185d254f21b2cc03b8d89374ad]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250905 19:58:17.814842 27638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ad0f154c598458fa48d8fd40ce46171. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.833842 27637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 650634b3e04147809fdde8cf72063c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.838956 27636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02a7a3e0f7a74cc2a1d2762c05eca777. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:17.873242 27696 tablet_bootstrap.cc:492] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f: Bootstrap starting.
I20250905 19:58:17.881738 27696 tablet_bootstrap.cc:654] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:17.875437 27695 tablet_bootstrap.cc:492] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8: Bootstrap starting.
I20250905 19:58:17.890445 27697 tablet_bootstrap.cc:492] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903: Bootstrap starting.
I20250905 19:58:17.901383 27695 tablet_bootstrap.cc:654] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:17.906649 27695 tablet_bootstrap.cc:492] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8: No bootstrap required, opened a new log
I20250905 19:58:17.907136 27695 ts_tablet_manager.cc:1397] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.004s
I20250905 19:58:17.908105 27696 tablet_bootstrap.cc:492] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f: No bootstrap required, opened a new log
I20250905 19:58:17.908589 27696 ts_tablet_manager.cc:1397] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.023s
I20250905 19:58:17.909806 27695 raft_consensus.cc:357] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:17.910611 27697 tablet_bootstrap.cc:654] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:17.910576 27695 raft_consensus.cc:383] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:17.911041 27695 raft_consensus.cc:738] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf97ab6f8494cdd8eb04ef0988476a8, State: Initialized, Role: FOLLOWER
I20250905 19:58:17.911290 27696 raft_consensus.cc:357] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:17.911942 27696 raft_consensus.cc:383] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:17.912360 27696 raft_consensus.cc:738] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Initialized, Role: FOLLOWER
I20250905 19:58:17.911866 27695 consensus_queue.cc:260] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:17.913007 27696 consensus_queue.cc:260] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:17.914552 27606 heartbeater.cc:499] Master 127.26.58.190:43327 was elected leader, sending a full tablet report...
I20250905 19:58:17.915359 27695 ts_tablet_manager.cc:1428] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20250905 19:58:17.916843 27695 tablet_bootstrap.cc:492] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8: Bootstrap starting.
I20250905 19:58:17.924502 27697 tablet_bootstrap.cc:492] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903: No bootstrap required, opened a new log
I20250905 19:58:17.924926 27695 tablet_bootstrap.cc:654] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:17.925077 27697 ts_tablet_manager.cc:1397] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903: Time spent bootstrapping tablet: real 0.035s user 0.011s sys 0.006s
I20250905 19:58:17.933099 27697 raft_consensus.cc:357] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:17.934175 27697 raft_consensus.cc:383] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:17.934625 27697 raft_consensus.cc:738] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15a3df6778944e9c8d1b663004a83903, State: Initialized, Role: FOLLOWER
I20250905 19:58:17.936314 27697 consensus_queue.cc:260] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:17.937608 27695 tablet_bootstrap.cc:492] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8: No bootstrap required, opened a new log
I20250905 19:58:17.938282 27695 ts_tablet_manager.cc:1397] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8: Time spent bootstrapping tablet: real 0.022s user 0.017s sys 0.001s
I20250905 19:58:17.943426 27695 raft_consensus.cc:357] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:17.944481 27695 raft_consensus.cc:383] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:17.944877 27695 raft_consensus.cc:738] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf97ab6f8494cdd8eb04ef0988476a8, State: Initialized, Role: FOLLOWER
I20250905 19:58:17.946058 27695 consensus_queue.cc:260] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:17.949434 27695 ts_tablet_manager.cc:1428] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250905 19:58:17.950881 27695 tablet_bootstrap.cc:492] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8: Bootstrap starting.
I20250905 19:58:17.960537 27695 tablet_bootstrap.cc:654] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:17.964171 27531 heartbeater.cc:499] Master 127.26.58.190:43327 was elected leader, sending a full tablet report...
I20250905 19:58:17.964854 27696 ts_tablet_manager.cc:1428] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f: Time spent starting tablet: real 0.056s user 0.015s sys 0.039s
I20250905 19:58:17.965770 27696 tablet_bootstrap.cc:492] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f: Bootstrap starting.
I20250905 19:58:17.972150 27696 tablet_bootstrap.cc:654] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:17.975747 27695 tablet_bootstrap.cc:492] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8: No bootstrap required, opened a new log
I20250905 19:58:17.976397 27695 ts_tablet_manager.cc:1397] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.011s
I20250905 19:58:17.979949 27680 heartbeater.cc:499] Master 127.26.58.190:43327 was elected leader, sending a full tablet report...
I20250905 19:58:17.981204 27697 ts_tablet_manager.cc:1428] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903: Time spent starting tablet: real 0.056s user 0.030s sys 0.018s
I20250905 19:58:17.979291 27695 raft_consensus.cc:357] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:17.982064 27695 raft_consensus.cc:383] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:17.982501 27697 tablet_bootstrap.cc:492] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903: Bootstrap starting.
I20250905 19:58:17.982378 27695 raft_consensus.cc:738] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf97ab6f8494cdd8eb04ef0988476a8, State: Initialized, Role: FOLLOWER
I20250905 19:58:17.983318 27695 consensus_queue.cc:260] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:17.986100 27695 ts_tablet_manager.cc:1428] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20250905 19:58:17.987123 27695 tablet_bootstrap.cc:492] T 1a6727b25590409b9402c5aee52a55a7 P 7bf97ab6f8494cdd8eb04ef0988476a8: Bootstrap starting.
I20250905 19:58:17.988777 27696 tablet_bootstrap.cc:492] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f: No bootstrap required, opened a new log
I20250905 19:58:17.989336 27696 ts_tablet_manager.cc:1397] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.005s
I20250905 19:58:17.991030 27697 tablet_bootstrap.cc:654] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:17.992321 27696 raft_consensus.cc:357] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:17.993108 27696 raft_consensus.cc:383] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:17.993309 27695 tablet_bootstrap.cc:654] T 1a6727b25590409b9402c5aee52a55a7 P 7bf97ab6f8494cdd8eb04ef0988476a8: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:17.993459 27696 raft_consensus.cc:738] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Initialized, Role: FOLLOWER
I20250905 19:58:17.994371 27696 consensus_queue.cc:260] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:17.996730 27696 ts_tablet_manager.cc:1428] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250905 19:58:17.997725 27696 tablet_bootstrap.cc:492] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f: Bootstrap starting.
I20250905 19:58:18.004122 27696 tablet_bootstrap.cc:654] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:18.007597 27697 tablet_bootstrap.cc:492] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903: No bootstrap required, opened a new log
I20250905 19:58:18.008249 27697 ts_tablet_manager.cc:1397] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.012s
I20250905 19:58:18.011091 27695 tablet_bootstrap.cc:492] T 1a6727b25590409b9402c5aee52a55a7 P 7bf97ab6f8494cdd8eb04ef0988476a8: No bootstrap required, opened a new log
I20250905 19:58:18.011684 27695 ts_tablet_manager.cc:1397] T 1a6727b25590409b9402c5aee52a55a7 P 7bf97ab6f8494cdd8eb04ef0988476a8: Time spent bootstrapping tablet: real 0.025s user 0.024s sys 0.000s
I20250905 19:58:18.011708 27697 raft_consensus.cc:357] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.012804 27697 raft_consensus.cc:383] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:18.013187 27697 raft_consensus.cc:738] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15a3df6778944e9c8d1b663004a83903, State: Initialized, Role: FOLLOWER
I20250905 19:58:18.014209 27697 consensus_queue.cc:260] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.015010 27695 raft_consensus.cc:357] T 1a6727b25590409b9402c5aee52a55a7 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } }
I20250905 19:58:18.015867 27695 raft_consensus.cc:383] T 1a6727b25590409b9402c5aee52a55a7 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:18.016230 27695 raft_consensus.cc:738] T 1a6727b25590409b9402c5aee52a55a7 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf97ab6f8494cdd8eb04ef0988476a8, State: Initialized, Role: FOLLOWER
I20250905 19:58:18.017284 27697 ts_tablet_manager.cc:1428] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s
I20250905 19:58:18.017001 27695 consensus_queue.cc:260] T 1a6727b25590409b9402c5aee52a55a7 P 7bf97ab6f8494cdd8eb04ef0988476a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } }
I20250905 19:58:18.018380 27697 tablet_bootstrap.cc:492] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903: Bootstrap starting.
I20250905 19:58:18.019678 27695 ts_tablet_manager.cc:1428] T 1a6727b25590409b9402c5aee52a55a7 P 7bf97ab6f8494cdd8eb04ef0988476a8: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250905 19:58:18.020705 27695 tablet_bootstrap.cc:492] T 66b1b69c203145e4bcc2cb26522c663f P 7bf97ab6f8494cdd8eb04ef0988476a8: Bootstrap starting.
I20250905 19:58:18.026706 27697 tablet_bootstrap.cc:654] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:18.026952 27695 tablet_bootstrap.cc:654] T 66b1b69c203145e4bcc2cb26522c663f P 7bf97ab6f8494cdd8eb04ef0988476a8: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:18.038712 27695 tablet_bootstrap.cc:492] T 66b1b69c203145e4bcc2cb26522c663f P 7bf97ab6f8494cdd8eb04ef0988476a8: No bootstrap required, opened a new log
I20250905 19:58:18.039330 27695 ts_tablet_manager.cc:1397] T 66b1b69c203145e4bcc2cb26522c663f P 7bf97ab6f8494cdd8eb04ef0988476a8: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.003s
I20250905 19:58:18.039572 27697 tablet_bootstrap.cc:492] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903: No bootstrap required, opened a new log
I20250905 19:58:18.040222 27697 ts_tablet_manager.cc:1397] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.011s
I20250905 19:58:18.042409 27695 raft_consensus.cc:357] T 66b1b69c203145e4bcc2cb26522c663f P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.043358 27695 raft_consensus.cc:383] T 66b1b69c203145e4bcc2cb26522c663f P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:18.043695 27695 raft_consensus.cc:738] T 66b1b69c203145e4bcc2cb26522c663f P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf97ab6f8494cdd8eb04ef0988476a8, State: Initialized, Role: FOLLOWER
I20250905 19:58:18.043741 27697 raft_consensus.cc:357] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.044513 27697 raft_consensus.cc:383] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:18.044760 27697 raft_consensus.cc:738] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15a3df6778944e9c8d1b663004a83903, State: Initialized, Role: FOLLOWER
I20250905 19:58:18.044749 27695 consensus_queue.cc:260] T 66b1b69c203145e4bcc2cb26522c663f P 7bf97ab6f8494cdd8eb04ef0988476a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.045740 27697 consensus_queue.cc:260] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.046379 27696 tablet_bootstrap.cc:492] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f: No bootstrap required, opened a new log
I20250905 19:58:18.046883 27696 ts_tablet_manager.cc:1397] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f: Time spent bootstrapping tablet: real 0.049s user 0.023s sys 0.018s
I20250905 19:58:18.047111 27695 ts_tablet_manager.cc:1428] T 66b1b69c203145e4bcc2cb26522c663f P 7bf97ab6f8494cdd8eb04ef0988476a8: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250905 19:58:18.048102 27695 tablet_bootstrap.cc:492] T 650634b3e04147809fdde8cf72063c78 P 7bf97ab6f8494cdd8eb04ef0988476a8: Bootstrap starting.
I20250905 19:58:18.048098 27697 ts_tablet_manager.cc:1428] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250905 19:58:18.049265 27697 tablet_bootstrap.cc:492] T 25d5dc671025465b90a24a0f20dae08f P 15a3df6778944e9c8d1b663004a83903: Bootstrap starting.
I20250905 19:58:18.049651 27696 raft_consensus.cc:357] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.050349 27696 raft_consensus.cc:383] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:18.050670 27696 raft_consensus.cc:738] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Initialized, Role: FOLLOWER
I20250905 19:58:18.051367 27696 consensus_queue.cc:260] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.053773 27696 ts_tablet_manager.cc:1428] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f: Time spent starting tablet: real 0.007s user 0.003s sys 0.001s
I20250905 19:58:18.054817 27696 tablet_bootstrap.cc:492] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f: Bootstrap starting.
I20250905 19:58:18.054970 27695 tablet_bootstrap.cc:654] T 650634b3e04147809fdde8cf72063c78 P 7bf97ab6f8494cdd8eb04ef0988476a8: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:18.055893 27697 tablet_bootstrap.cc:654] T 25d5dc671025465b90a24a0f20dae08f P 15a3df6778944e9c8d1b663004a83903: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:18.062307 27696 tablet_bootstrap.cc:654] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:18.066780 27695 tablet_bootstrap.cc:492] T 650634b3e04147809fdde8cf72063c78 P 7bf97ab6f8494cdd8eb04ef0988476a8: No bootstrap required, opened a new log
I20250905 19:58:18.067281 27695 ts_tablet_manager.cc:1397] T 650634b3e04147809fdde8cf72063c78 P 7bf97ab6f8494cdd8eb04ef0988476a8: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.004s
I20250905 19:58:18.067457 27697 tablet_bootstrap.cc:492] T 25d5dc671025465b90a24a0f20dae08f P 15a3df6778944e9c8d1b663004a83903: No bootstrap required, opened a new log
I20250905 19:58:18.068071 27697 ts_tablet_manager.cc:1397] T 25d5dc671025465b90a24a0f20dae08f P 15a3df6778944e9c8d1b663004a83903: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.012s
I20250905 19:58:18.069945 27695 raft_consensus.cc:357] T 650634b3e04147809fdde8cf72063c78 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.070741 27695 raft_consensus.cc:383] T 650634b3e04147809fdde8cf72063c78 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:18.071025 27695 raft_consensus.cc:738] T 650634b3e04147809fdde8cf72063c78 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf97ab6f8494cdd8eb04ef0988476a8, State: Initialized, Role: FOLLOWER
I20250905 19:58:18.071554 27697 raft_consensus.cc:357] T 25d5dc671025465b90a24a0f20dae08f P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.071900 27695 consensus_queue.cc:260] T 650634b3e04147809fdde8cf72063c78 P 7bf97ab6f8494cdd8eb04ef0988476a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.072523 27697 raft_consensus.cc:383] T 25d5dc671025465b90a24a0f20dae08f P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:18.073508 27697 raft_consensus.cc:738] T 25d5dc671025465b90a24a0f20dae08f P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15a3df6778944e9c8d1b663004a83903, State: Initialized, Role: FOLLOWER
I20250905 19:58:18.074407 27702 raft_consensus.cc:491] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:18.074472 27697 consensus_queue.cc:260] T 25d5dc671025465b90a24a0f20dae08f P 15a3df6778944e9c8d1b663004a83903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.074998 27702 raft_consensus.cc:513] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.075354 27695 ts_tablet_manager.cc:1428] T 650634b3e04147809fdde8cf72063c78 P 7bf97ab6f8494cdd8eb04ef0988476a8: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250905 19:58:18.076424 27695 tablet_bootstrap.cc:492] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8: Bootstrap starting.
I20250905 19:58:18.080595 27702 leader_election.cc:290] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929), 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:18.083487 27702 raft_consensus.cc:491] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:18.083918 27702 raft_consensus.cc:513] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.085246 27696 tablet_bootstrap.cc:492] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f: No bootstrap required, opened a new log
I20250905 19:58:18.085328 27702 leader_election.cc:290] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929), 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:18.080950 27697 ts_tablet_manager.cc:1428] T 25d5dc671025465b90a24a0f20dae08f P 15a3df6778944e9c8d1b663004a83903: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20250905 19:58:18.082796 27695 tablet_bootstrap.cc:654] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:18.085835 27696 ts_tablet_manager.cc:1397] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.017s
I20250905 19:58:18.086659 27697 tablet_bootstrap.cc:492] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903: Bootstrap starting.
I20250905 19:58:18.088649 27696 raft_consensus.cc:357] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.089349 27696 raft_consensus.cc:383] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:18.089650 27696 raft_consensus.cc:738] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Initialized, Role: FOLLOWER
I20250905 19:58:18.090451 27696 consensus_queue.cc:260] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.092752 27696 ts_tablet_manager.cc:1428] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20250905 19:58:18.103008 27696 tablet_bootstrap.cc:492] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 025319773a4247b99829aa83dd002b0f: Bootstrap starting.
I20250905 19:58:18.109503 27695 tablet_bootstrap.cc:492] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8: No bootstrap required, opened a new log
I20250905 19:58:18.109787 27655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66b1b69c203145e4bcc2cb26522c663f" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15a3df6778944e9c8d1b663004a83903" is_pre_election: true
I20250905 19:58:18.110234 27697 tablet_bootstrap.cc:654] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:18.110347 27696 tablet_bootstrap.cc:654] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 025319773a4247b99829aa83dd002b0f: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:18.110167 27695 ts_tablet_manager.cc:1397] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.022s
I20250905 19:58:18.117413 27581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66b1b69c203145e4bcc2cb26522c663f" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" is_pre_election: true
I20250905 19:58:18.118176 27581 raft_consensus.cc:2466] T 66b1b69c203145e4bcc2cb26522c663f P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 025319773a4247b99829aa83dd002b0f in term 0.
I20250905 19:58:18.119666 27580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d5dc671025465b90a24a0f20dae08f" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" is_pre_election: true
I20250905 19:58:18.119951 27695 raft_consensus.cc:357] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.121492 27695 raft_consensus.cc:383] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:18.121953 27695 raft_consensus.cc:738] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf97ab6f8494cdd8eb04ef0988476a8, State: Initialized, Role: FOLLOWER
I20250905 19:58:18.123195 27695 consensus_queue.cc:260] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.126381 27472 leader_election.cc:304] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025319773a4247b99829aa83dd002b0f, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters:
I20250905 19:58:18.127004 27695 ts_tablet_manager.cc:1428] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8: Time spent starting tablet: real 0.010s user 0.008s sys 0.001s
W20250905 19:58:18.127377 27472 leader_election.cc:343] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:18.128232 27702 raft_consensus.cc:2802] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:18.128715 27702 raft_consensus.cc:491] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:18.129245 27702 raft_consensus.cc:3058] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.110679 27655 raft_consensus.cc:2466] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 025319773a4247b99829aa83dd002b0f in term 0.
I20250905 19:58:18.111335 27654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d5dc671025465b90a24a0f20dae08f" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15a3df6778944e9c8d1b663004a83903" is_pre_election: true
I20250905 19:58:18.134230 27654 raft_consensus.cc:2466] T 25d5dc671025465b90a24a0f20dae08f P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 025319773a4247b99829aa83dd002b0f in term 0.
I20250905 19:58:18.135982 27471 leader_election.cc:304] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 025319773a4247b99829aa83dd002b0f, 15a3df6778944e9c8d1b663004a83903; no voters: 7bf97ab6f8494cdd8eb04ef0988476a8
I20250905 19:58:18.137434 27717 raft_consensus.cc:2802] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:18.137827 27717 raft_consensus.cc:491] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:18.138159 27717 raft_consensus.cc:3058] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.139740 27704 raft_consensus.cc:491] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:18.140125 27704 raft_consensus.cc:513] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.142073 27704 leader_election.cc:290] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725), 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929)
I20250905 19:58:18.144147 27696 tablet_bootstrap.cc:492] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 025319773a4247b99829aa83dd002b0f: No bootstrap required, opened a new log
I20250905 19:58:18.144762 27696 ts_tablet_manager.cc:1397] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 025319773a4247b99829aa83dd002b0f: Time spent bootstrapping tablet: real 0.042s user 0.015s sys 0.018s
I20250905 19:58:18.145467 27700 raft_consensus.cc:491] T 1a6727b25590409b9402c5aee52a55a7 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:18.145901 27700 raft_consensus.cc:513] T 1a6727b25590409b9402c5aee52a55a7 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } }
I20250905 19:58:18.146258 27717 raft_consensus.cc:513] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.147804 27696 raft_consensus.cc:357] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.148535 27696 raft_consensus.cc:383] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:18.148772 27700 leader_election.cc:290] T 1a6727b25590409b9402c5aee52a55a7 P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617), 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725)
I20250905 19:58:18.148804 27696 raft_consensus.cc:738] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Initialized, Role: FOLLOWER
I20250905 19:58:18.150153 27696 consensus_queue.cc:260] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 025319773a4247b99829aa83dd002b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.161134 27696 ts_tablet_manager.cc:1428] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 025319773a4247b99829aa83dd002b0f: Time spent starting tablet: real 0.016s user 0.013s sys 0.001s
I20250905 19:58:18.165772 27696 tablet_bootstrap.cc:492] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f: Bootstrap starting.
I20250905 19:58:18.136932 27702 raft_consensus.cc:513] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.169826 27580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66b1b69c203145e4bcc2cb26522c663f" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
I20250905 19:58:18.170608 27580 raft_consensus.cc:3058] T 66b1b69c203145e4bcc2cb26522c663f P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.170724 27702 leader_election.cc:290] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 election: Requested vote from peers 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929), 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:18.128695 27695 tablet_bootstrap.cc:492] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8: Bootstrap starting.
I20250905 19:58:18.172966 27696 tablet_bootstrap.cc:654] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:18.174815 27654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66b1b69c203145e4bcc2cb26522c663f" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15a3df6778944e9c8d1b663004a83903"
I20250905 19:58:18.175446 27654 raft_consensus.cc:3058] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.180723 27581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d5dc671025465b90a24a0f20dae08f" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
I20250905 19:58:18.181010 27717 leader_election.cc:290] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 election: Requested vote from peers 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929), 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:18.182946 27655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d5dc671025465b90a24a0f20dae08f" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15a3df6778944e9c8d1b663004a83903"
I20250905 19:58:18.184978 27655 raft_consensus.cc:3058] T 25d5dc671025465b90a24a0f20dae08f P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.188236 27695 tablet_bootstrap.cc:654] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8: Neither blocks nor log segments found. Creating new log.
W20250905 19:58:18.193074 27472 leader_election.cc:343] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:18.197019 27655 raft_consensus.cc:2466] T 25d5dc671025465b90a24a0f20dae08f P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 025319773a4247b99829aa83dd002b0f in term 1.
I20250905 19:58:18.198163 27471 leader_election.cc:304] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 025319773a4247b99829aa83dd002b0f, 15a3df6778944e9c8d1b663004a83903; no voters: 7bf97ab6f8494cdd8eb04ef0988476a8
I20250905 19:58:18.201161 27696 tablet_bootstrap.cc:492] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f: No bootstrap required, opened a new log
I20250905 19:58:18.201238 27717 raft_consensus.cc:2802] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:18.201551 27697 tablet_bootstrap.cc:492] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903: No bootstrap required, opened a new log
I20250905 19:58:18.201718 27696 ts_tablet_manager.cc:1397] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f: Time spent bootstrapping tablet: real 0.036s user 0.020s sys 0.007s
I20250905 19:58:18.201701 27717 raft_consensus.cc:695] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f [term 1 LEADER]: Becoming Leader. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Running, Role: LEADER
I20250905 19:58:18.201974 27697 ts_tablet_manager.cc:1397] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903: Time spent bootstrapping tablet: real 0.116s user 0.029s sys 0.017s
I20250905 19:58:18.204382 27697 raft_consensus.cc:357] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.204701 27696 raft_consensus.cc:357] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.205128 27697 raft_consensus.cc:383] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:18.205384 27696 raft_consensus.cc:383] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:18.205562 27697 raft_consensus.cc:738] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15a3df6778944e9c8d1b663004a83903, State: Initialized, Role: FOLLOWER
I20250905 19:58:18.205781 27696 raft_consensus.cc:738] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Initialized, Role: FOLLOWER
I20250905 19:58:18.206526 27697 consensus_queue.cc:260] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.206545 27696 consensus_queue.cc:260] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.209156 27696 ts_tablet_manager.cc:1428] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250905 19:58:18.210107 27696 tablet_bootstrap.cc:492] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f: Bootstrap starting.
I20250905 19:58:18.211030 27580 raft_consensus.cc:2466] T 66b1b69c203145e4bcc2cb26522c663f P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 025319773a4247b99829aa83dd002b0f in term 1.
I20250905 19:58:18.213835 27655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a6727b25590409b9402c5aee52a55a7" candidate_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15a3df6778944e9c8d1b663004a83903" is_pre_election: true
I20250905 19:58:18.221477 27506 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccec5ef5fb0d4383a93e6cf0514aefb8" candidate_uuid: "15a3df6778944e9c8d1b663004a83903" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "025319773a4247b99829aa83dd002b0f" is_pre_election: true
I20250905 19:58:18.222589 27695 tablet_bootstrap.cc:492] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8: No bootstrap required, opened a new log
W20250905 19:58:18.222507 27546 leader_election.cc:343] T 1a6727b25590409b9402c5aee52a55a7 P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:18.223052 27695 ts_tablet_manager.cc:1397] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8: Time spent bootstrapping tablet: real 0.095s user 0.007s sys 0.006s
W20250905 19:58:18.214846 27452 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250905 19:58:18.224100 27472 leader_election.cc:304] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025319773a4247b99829aa83dd002b0f, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters:
I20250905 19:58:18.215523 27697 ts_tablet_manager.cc:1428] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20250905 19:58:18.225060 27725 raft_consensus.cc:2802] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:18.225701 27725 raft_consensus.cc:695] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [term 1 LEADER]: Becoming Leader. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Running, Role: LEADER
I20250905 19:58:18.225631 27695 raft_consensus.cc:357] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.226326 27695 raft_consensus.cc:383] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:18.226621 27695 raft_consensus.cc:738] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf97ab6f8494cdd8eb04ef0988476a8, State: Initialized, Role: FOLLOWER
I20250905 19:58:18.226627 27725 consensus_queue.cc:237] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.227262 27695 consensus_queue.cc:260] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.232484 27697 tablet_bootstrap.cc:492] T 7fa1390a6ad647498e7b9459277a60ac P 15a3df6778944e9c8d1b663004a83903: Bootstrap starting.
I20250905 19:58:18.216064 27717 consensus_queue.cc:237] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.216420 27696 tablet_bootstrap.cc:654] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:18.232899 27506 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a6727b25590409b9402c5aee52a55a7" candidate_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "025319773a4247b99829aa83dd002b0f" is_pre_election: true
W20250905 19:58:18.234495 27547 leader_election.cc:343] T 1a6727b25590409b9402c5aee52a55a7 P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:18.235008 27547 leader_election.cc:304] T 1a6727b25590409b9402c5aee52a55a7 P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7bf97ab6f8494cdd8eb04ef0988476a8; no voters: 025319773a4247b99829aa83dd002b0f, 15a3df6778944e9c8d1b663004a83903
I20250905 19:58:18.216439 27700 raft_consensus.cc:491] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:18.236326 27700 raft_consensus.cc:513] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.216676 27654 raft_consensus.cc:2466] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 025319773a4247b99829aa83dd002b0f in term 1.
W20250905 19:58:18.238245 27621 leader_election.cc:343] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:18.238817 27700 leader_election.cc:290] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725), 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:18.242794 27700 raft_consensus.cc:2747] T 1a6727b25590409b9402c5aee52a55a7 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:18.243423 27700 raft_consensus.cc:491] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:18.243858 27700 raft_consensus.cc:513] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.244609 27695 ts_tablet_manager.cc:1428] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8: Time spent starting tablet: real 0.021s user 0.010s sys 0.001s
I20250905 19:58:18.240434 27697 tablet_bootstrap.cc:654] T 7fa1390a6ad647498e7b9459277a60ac P 15a3df6778944e9c8d1b663004a83903: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:18.246683 27700 leader_election.cc:290] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725), 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:18.246862 27695 tablet_bootstrap.cc:492] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8: Bootstrap starting.
I20250905 19:58:18.248651 27654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02a7a3e0f7a74cc2a1d2762c05eca777" candidate_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15a3df6778944e9c8d1b663004a83903" is_pre_election: true
I20250905 19:58:18.249853 27655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3ad0f154c598458fa48d8fd40ce46171" candidate_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15a3df6778944e9c8d1b663004a83903" is_pre_election: true
W20250905 19:58:18.251139 27546 leader_election.cc:343] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:58:18.252121 27546 leader_election.cc:343] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:18.253844 27695 tablet_bootstrap.cc:654] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:18.258912 27401 catalog_manager.cc:5582] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 025319773a4247b99829aa83dd002b0f (127.26.58.129). New cstate: current_term: 1 leader_uuid: "025319773a4247b99829aa83dd002b0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:18.260187 27506 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02a7a3e0f7a74cc2a1d2762c05eca777" candidate_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "025319773a4247b99829aa83dd002b0f" is_pre_election: true
I20250905 19:58:18.261010 27506 raft_consensus.cc:2466] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bf97ab6f8494cdd8eb04ef0988476a8 in term 0.
I20250905 19:58:18.262220 27505 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3ad0f154c598458fa48d8fd40ce46171" candidate_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "025319773a4247b99829aa83dd002b0f" is_pre_election: true
I20250905 19:58:18.264343 27547 leader_election.cc:304] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 025319773a4247b99829aa83dd002b0f, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters: 15a3df6778944e9c8d1b663004a83903
W20250905 19:58:18.265966 27547 leader_election.cc:343] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:18.266640 27547 leader_election.cc:304] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7bf97ab6f8494cdd8eb04ef0988476a8; no voters: 025319773a4247b99829aa83dd002b0f, 15a3df6778944e9c8d1b663004a83903
I20250905 19:58:18.267913 27700 raft_consensus.cc:2802] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:18.268481 27700 raft_consensus.cc:491] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:18.268900 27700 raft_consensus.cc:3058] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.270326 27695 tablet_bootstrap.cc:492] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8: No bootstrap required, opened a new log
I20250905 19:58:18.270849 27695 ts_tablet_manager.cc:1397] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.000s
I20250905 19:58:18.273653 27704 raft_consensus.cc:491] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:18.274574 27733 raft_consensus.cc:2747] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:18.273491 27695 raft_consensus.cc:357] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.278963 27695 raft_consensus.cc:383] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:18.274364 27704 raft_consensus.cc:513] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.279276 27695 raft_consensus.cc:738] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf97ab6f8494cdd8eb04ef0988476a8, State: Initialized, Role: FOLLOWER
I20250905 19:58:18.281360 27704 leader_election.cc:290] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725), 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929)
I20250905 19:58:18.281368 27580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccec5ef5fb0d4383a93e6cf0514aefb8" candidate_uuid: "15a3df6778944e9c8d1b663004a83903" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" is_pre_election: true
I20250905 19:58:18.281085 27695 consensus_queue.cc:260] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.282567 27580 raft_consensus.cc:2466] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15a3df6778944e9c8d1b663004a83903 in term 0.
I20250905 19:58:18.283443 27505 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa10aea786b5462c82a92fc0f9bff996" candidate_uuid: "15a3df6778944e9c8d1b663004a83903" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "025319773a4247b99829aa83dd002b0f" is_pre_election: true
I20250905 19:58:18.284055 27505 raft_consensus.cc:2466] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15a3df6778944e9c8d1b663004a83903 in term 0.
I20250905 19:58:18.284004 27695 ts_tablet_manager.cc:1428] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20250905 19:58:18.286710 27621 leader_election.cc:304] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025319773a4247b99829aa83dd002b0f, 15a3df6778944e9c8d1b663004a83903; no voters:
I20250905 19:58:18.287058 27695 tablet_bootstrap.cc:492] T 25d5dc671025465b90a24a0f20dae08f P 7bf97ab6f8494cdd8eb04ef0988476a8: Bootstrap starting.
I20250905 19:58:18.291429 27700 raft_consensus.cc:513] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.292201 27704 raft_consensus.cc:2802] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:18.286738 27580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa10aea786b5462c82a92fc0f9bff996" candidate_uuid: "15a3df6778944e9c8d1b663004a83903" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" is_pre_election: true
I20250905 19:58:18.293049 27704 raft_consensus.cc:491] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:18.292337 27621 leader_election.cc:304] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 15a3df6778944e9c8d1b663004a83903, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters: 025319773a4247b99829aa83dd002b0f
I20250905 19:58:18.293814 27695 tablet_bootstrap.cc:654] T 25d5dc671025465b90a24a0f20dae08f P 7bf97ab6f8494cdd8eb04ef0988476a8: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:18.293569 27704 raft_consensus.cc:3058] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.296780 27696 tablet_bootstrap.cc:492] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f: No bootstrap required, opened a new log
I20250905 19:58:18.296777 27505 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02a7a3e0f7a74cc2a1d2762c05eca777" candidate_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "025319773a4247b99829aa83dd002b0f"
I20250905 19:58:18.297410 27696 ts_tablet_manager.cc:1397] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f: Time spent bootstrapping tablet: real 0.087s user 0.017s sys 0.019s
I20250905 19:58:18.297466 27505 raft_consensus.cc:3058] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.293553 27580 raft_consensus.cc:2466] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15a3df6778944e9c8d1b663004a83903 in term 0.
I20250905 19:58:18.300557 27696 raft_consensus.cc:357] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.301262 27696 raft_consensus.cc:383] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:18.301614 27696 raft_consensus.cc:738] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Initialized, Role: FOLLOWER
I20250905 19:58:18.303031 27738 raft_consensus.cc:2802] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:18.303442 27738 raft_consensus.cc:491] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:18.303781 27738 raft_consensus.cc:3058] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.305140 27696 consensus_queue.cc:260] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.305743 27505 raft_consensus.cc:2466] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 025319773a4247b99829aa83dd002b0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bf97ab6f8494cdd8eb04ef0988476a8 in term 1.
I20250905 19:58:18.307138 27547 leader_election.cc:304] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025319773a4247b99829aa83dd002b0f, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters:
I20250905 19:58:18.308071 27696 ts_tablet_manager.cc:1428] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20250905 19:58:18.308823 27733 raft_consensus.cc:2802] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:18.309125 27696 tablet_bootstrap.cc:492] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f: Bootstrap starting.
I20250905 19:58:18.309490 27733 raft_consensus.cc:695] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7bf97ab6f8494cdd8eb04ef0988476a8, State: Running, Role: LEADER
I20250905 19:58:18.310458 27733 consensus_queue.cc:237] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.314095 27700 leader_election.cc:290] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 1 election: Requested vote from peers 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725), 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:18.318496 27655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02a7a3e0f7a74cc2a1d2762c05eca777" candidate_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15a3df6778944e9c8d1b663004a83903"
W20250905 19:58:18.319849 27546 leader_election.cc:343] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:18.321028 27700 raft_consensus.cc:491] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:18.321708 27717 raft_consensus.cc:491] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:18.321612 27700 raft_consensus.cc:513] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.322190 27717 raft_consensus.cc:513] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.323827 27697 tablet_bootstrap.cc:492] T 7fa1390a6ad647498e7b9459277a60ac P 15a3df6778944e9c8d1b663004a83903: No bootstrap required, opened a new log
I20250905 19:58:18.324057 27717 leader_election.cc:290] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929), 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:18.324855 27505 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eb30fb918af423aba649dba5b74f749" candidate_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "025319773a4247b99829aa83dd002b0f" is_pre_election: true
I20250905 19:58:18.325369 27704 raft_consensus.cc:513] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.326064 27580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "650634b3e04147809fdde8cf72063c78" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" is_pre_election: true
I20250905 19:58:18.326721 27580 raft_consensus.cc:2466] T 650634b3e04147809fdde8cf72063c78 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 025319773a4247b99829aa83dd002b0f in term 0.
I20250905 19:58:18.326747 27655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "650634b3e04147809fdde8cf72063c78" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15a3df6778944e9c8d1b663004a83903" is_pre_election: true
I20250905 19:58:18.327922 27472 leader_election.cc:304] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025319773a4247b99829aa83dd002b0f, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters:
W20250905 19:58:18.328445 27547 leader_election.cc:343] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:18.328795 27717 raft_consensus.cc:2802] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250905 19:58:18.329052 27471 leader_election.cc:343] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:18.329190 27717 raft_consensus.cc:491] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:18.329602 27717 raft_consensus.cc:3058] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.330920 27697 ts_tablet_manager.cc:1397] T 7fa1390a6ad647498e7b9459277a60ac P 15a3df6778944e9c8d1b663004a83903: Time spent bootstrapping tablet: real 0.099s user 0.041s sys 0.030s
I20250905 19:58:18.333840 27700 leader_election.cc:290] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725), 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:18.334195 27697 raft_consensus.cc:357] T 7fa1390a6ad647498e7b9459277a60ac P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.335018 27697 raft_consensus.cc:383] T 7fa1390a6ad647498e7b9459277a60ac P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:18.335314 27697 raft_consensus.cc:738] T 7fa1390a6ad647498e7b9459277a60ac P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15a3df6778944e9c8d1b663004a83903, State: Initialized, Role: FOLLOWER
I20250905 19:58:18.336887 27717 raft_consensus.cc:513] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.338317 27704 leader_election.cc:290] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [CANDIDATE]: Term 1 election: Requested vote from peers 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725), 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929)
I20250905 19:58:18.336097 27697 consensus_queue.cc:260] T 7fa1390a6ad647498e7b9459277a60ac P 15a3df6778944e9c8d1b663004a83903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.339373 27655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eb30fb918af423aba649dba5b74f749" candidate_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15a3df6778944e9c8d1b663004a83903" is_pre_election: true
I20250905 19:58:18.339588 27654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "650634b3e04147809fdde8cf72063c78" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15a3df6778944e9c8d1b663004a83903"
I20250905 19:58:18.339969 27655 raft_consensus.cc:2466] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bf97ab6f8494cdd8eb04ef0988476a8 in term 0.
W20250905 19:58:18.341082 27471 leader_election.cc:343] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:18.340993 27546 leader_election.cc:304] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 15a3df6778944e9c8d1b663004a83903, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters: 025319773a4247b99829aa83dd002b0f
I20250905 19:58:18.341434 27697 ts_tablet_manager.cc:1428] T 7fa1390a6ad647498e7b9459277a60ac P 15a3df6778944e9c8d1b663004a83903: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20250905 19:58:18.342414 27697 tablet_bootstrap.cc:492] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903: Bootstrap starting.
I20250905 19:58:18.343019 27696 tablet_bootstrap.cc:654] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:18.342857 27580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa10aea786b5462c82a92fc0f9bff996" candidate_uuid: "15a3df6778944e9c8d1b663004a83903" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
I20250905 19:58:18.343442 27580 raft_consensus.cc:3058] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.338414 27717 leader_election.cc:290] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 election: Requested vote from peers 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929), 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:18.345876 27738 raft_consensus.cc:513] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.346819 27581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "650634b3e04147809fdde8cf72063c78" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
I20250905 19:58:18.347374 27581 raft_consensus.cc:3058] T 650634b3e04147809fdde8cf72063c78 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.347973 27738 leader_election.cc:290] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [CANDIDATE]: Term 1 election: Requested vote from peers 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725), 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929)
I20250905 19:58:18.349499 27505 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa10aea786b5462c82a92fc0f9bff996" candidate_uuid: "15a3df6778944e9c8d1b663004a83903" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "025319773a4247b99829aa83dd002b0f"
I20250905 19:58:18.350253 27505 raft_consensus.cc:3058] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.353114 27581 raft_consensus.cc:2466] T 650634b3e04147809fdde8cf72063c78 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 025319773a4247b99829aa83dd002b0f in term 1.
I20250905 19:58:18.353452 27697 tablet_bootstrap.cc:654] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:18.354010 27506 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccec5ef5fb0d4383a93e6cf0514aefb8" candidate_uuid: "15a3df6778944e9c8d1b663004a83903" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "025319773a4247b99829aa83dd002b0f"
I20250905 19:58:18.354259 27472 leader_election.cc:304] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 025319773a4247b99829aa83dd002b0f, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters: 15a3df6778944e9c8d1b663004a83903
I20250905 19:58:18.355430 27717 raft_consensus.cc:2802] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:18.355897 27717 raft_consensus.cc:695] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [term 1 LEADER]: Becoming Leader. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Running, Role: LEADER
I20250905 19:58:18.357533 27717 consensus_queue.cc:237] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.359910 27581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccec5ef5fb0d4383a93e6cf0514aefb8" candidate_uuid: "15a3df6778944e9c8d1b663004a83903" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
I20250905 19:58:18.360162 27700 raft_consensus.cc:2802] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:18.360742 27700 raft_consensus.cc:491] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:18.361223 27580 raft_consensus.cc:2466] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15a3df6778944e9c8d1b663004a83903 in term 1.
I20250905 19:58:18.361792 27700 raft_consensus.cc:3058] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.365476 27695 tablet_bootstrap.cc:492] T 25d5dc671025465b90a24a0f20dae08f P 7bf97ab6f8494cdd8eb04ef0988476a8: No bootstrap required, opened a new log
I20250905 19:58:18.365983 27695 ts_tablet_manager.cc:1397] T 25d5dc671025465b90a24a0f20dae08f P 7bf97ab6f8494cdd8eb04ef0988476a8: Time spent bootstrapping tablet: real 0.079s user 0.009s sys 0.002s
I20250905 19:58:18.360743 27581 raft_consensus.cc:3058] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.369220 27695 raft_consensus.cc:357] T 25d5dc671025465b90a24a0f20dae08f P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.369937 27695 raft_consensus.cc:383] T 25d5dc671025465b90a24a0f20dae08f P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:18.370847 27695 raft_consensus.cc:738] T 25d5dc671025465b90a24a0f20dae08f P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf97ab6f8494cdd8eb04ef0988476a8, State: Initialized, Role: FOLLOWER
I20250905 19:58:18.371642 27695 consensus_queue.cc:260] T 25d5dc671025465b90a24a0f20dae08f P 7bf97ab6f8494cdd8eb04ef0988476a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.371263 27400 catalog_manager.cc:5582] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:18.374047 27695 ts_tablet_manager.cc:1428] T 25d5dc671025465b90a24a0f20dae08f P 7bf97ab6f8494cdd8eb04ef0988476a8: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
W20250905 19:58:18.366645 27621 leader_election.cc:343] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:18.378754 27733 raft_consensus.cc:491] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:18.379266 27733 raft_consensus.cc:513] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.380484 27621 leader_election.cc:304] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15a3df6778944e9c8d1b663004a83903, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters:
I20250905 19:58:18.381341 27738 raft_consensus.cc:2802] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:18.381451 27733 leader_election.cc:290] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725), 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:18.382745 27581 raft_consensus.cc:2466] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15a3df6778944e9c8d1b663004a83903 in term 1.
I20250905 19:58:18.383853 27655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fa1390a6ad647498e7b9459277a60ac" candidate_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15a3df6778944e9c8d1b663004a83903" is_pre_election: true
I20250905 19:58:18.383860 27506 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fa1390a6ad647498e7b9459277a60ac" candidate_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "025319773a4247b99829aa83dd002b0f" is_pre_election: true
I20250905 19:58:18.381913 27738 raft_consensus.cc:695] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [term 1 LEADER]: Becoming Leader. State: Replica: 15a3df6778944e9c8d1b663004a83903, State: Running, Role: LEADER
I20250905 19:58:18.384640 27506 raft_consensus.cc:2466] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bf97ab6f8494cdd8eb04ef0988476a8 in term 0.
I20250905 19:58:18.385260 27621 leader_election.cc:304] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 15a3df6778944e9c8d1b663004a83903, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters: 025319773a4247b99829aa83dd002b0f
I20250905 19:58:18.386111 27704 raft_consensus.cc:2802] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:18.385680 27738 consensus_queue.cc:237] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.384640 27655 raft_consensus.cc:2466] T 7fa1390a6ad647498e7b9459277a60ac P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bf97ab6f8494cdd8eb04ef0988476a8 in term 0.
I20250905 19:58:18.385982 27547 leader_election.cc:304] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025319773a4247b99829aa83dd002b0f, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters:
I20250905 19:58:18.394122 27733 raft_consensus.cc:2802] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:18.394985 27733 raft_consensus.cc:491] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:18.395323 27733 raft_consensus.cc:3058] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.398114 27704 raft_consensus.cc:695] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [term 1 LEADER]: Becoming Leader. State: Replica: 15a3df6778944e9c8d1b663004a83903, State: Running, Role: LEADER
I20250905 19:58:18.401014 27505 raft_consensus.cc:2466] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15a3df6778944e9c8d1b663004a83903 in term 1.
I20250905 19:58:18.400907 27704 consensus_queue.cc:237] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.403136 27696 tablet_bootstrap.cc:492] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f: No bootstrap required, opened a new log
I20250905 19:58:18.403502 27696 ts_tablet_manager.cc:1397] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f: Time spent bootstrapping tablet: real 0.095s user 0.005s sys 0.011s
I20250905 19:58:18.405206 27697 tablet_bootstrap.cc:492] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903: No bootstrap required, opened a new log
I20250905 19:58:18.408836 27700 raft_consensus.cc:513] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.407071 27697 ts_tablet_manager.cc:1397] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903: Time spent bootstrapping tablet: real 0.065s user 0.005s sys 0.011s
I20250905 19:58:18.410061 27696 raft_consensus.cc:357] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.411068 27696 raft_consensus.cc:383] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:18.411442 27696 raft_consensus.cc:738] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Initialized, Role: FOLLOWER
I20250905 19:58:18.412237 27696 consensus_queue.cc:260] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.416723 27700 leader_election.cc:290] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 1 election: Requested vote from peers 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725), 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:18.420358 27655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eb30fb918af423aba649dba5b74f749" candidate_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15a3df6778944e9c8d1b663004a83903"
I20250905 19:58:18.420965 27655 raft_consensus.cc:3058] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.423990 27733 raft_consensus.cc:513] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.424402 27400 catalog_manager.cc:5582] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 025319773a4247b99829aa83dd002b0f (127.26.58.129). New cstate: current_term: 1 leader_uuid: "025319773a4247b99829aa83dd002b0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:18.426644 27400 catalog_manager.cc:5582] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 025319773a4247b99829aa83dd002b0f (127.26.58.129). New cstate: current_term: 1 leader_uuid: "025319773a4247b99829aa83dd002b0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:18.427699 27505 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eb30fb918af423aba649dba5b74f749" candidate_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "025319773a4247b99829aa83dd002b0f"
I20250905 19:58:18.428838 27506 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fa1390a6ad647498e7b9459277a60ac" candidate_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "025319773a4247b99829aa83dd002b0f"
I20250905 19:58:18.429368 27506 raft_consensus.cc:3058] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.430538 27697 raft_consensus.cc:357] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } }
I20250905 19:58:18.431510 27697 raft_consensus.cc:383] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:18.431851 27697 raft_consensus.cc:738] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15a3df6778944e9c8d1b663004a83903, State: Initialized, Role: FOLLOWER
I20250905 19:58:18.432693 27697 consensus_queue.cc:260] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } }
W20250905 19:58:18.434227 27547 leader_election.cc:343] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:18.436086 27697 ts_tablet_manager.cc:1428] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903: Time spent starting tablet: real 0.018s user 0.004s sys 0.003s
I20250905 19:58:18.437089 27697 tablet_bootstrap.cc:492] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 15a3df6778944e9c8d1b663004a83903: Bootstrap starting.
I20250905 19:58:18.440506 27506 raft_consensus.cc:2466] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bf97ab6f8494cdd8eb04ef0988476a8 in term 1.
I20250905 19:58:18.442875 27655 raft_consensus.cc:2466] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bf97ab6f8494cdd8eb04ef0988476a8 in term 1.
I20250905 19:58:18.444363 27697 tablet_bootstrap.cc:654] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 15a3df6778944e9c8d1b663004a83903: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:18.446651 27733 leader_election.cc:290] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 1 election: Requested vote from peers 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725), 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:18.445778 27696 ts_tablet_manager.cc:1428] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f: Time spent starting tablet: real 0.042s user 0.006s sys 0.000s
I20250905 19:58:18.447608 27546 leader_election.cc:304] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 15a3df6778944e9c8d1b663004a83903, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters: 025319773a4247b99829aa83dd002b0f
I20250905 19:58:18.445827 27401 catalog_manager.cc:5582] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15a3df6778944e9c8d1b663004a83903 (127.26.58.131). New cstate: current_term: 1 leader_uuid: "15a3df6778944e9c8d1b663004a83903" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } health_report { overall_health: HEALTHY } } }
I20250905 19:58:18.448740 27655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fa1390a6ad647498e7b9459277a60ac" candidate_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15a3df6778944e9c8d1b663004a83903"
I20250905 19:58:18.449198 27696 tablet_bootstrap.cc:492] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f: Bootstrap starting.
I20250905 19:58:18.451807 27547 leader_election.cc:304] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025319773a4247b99829aa83dd002b0f, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters:
I20250905 19:58:18.452695 27700 raft_consensus.cc:2802] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:18.453169 27700 raft_consensus.cc:695] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7bf97ab6f8494cdd8eb04ef0988476a8, State: Running, Role: LEADER
I20250905 19:58:18.453704 27655 raft_consensus.cc:3058] T 7fa1390a6ad647498e7b9459277a60ac P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.449388 27733 raft_consensus.cc:2802] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:18.453977 27700 consensus_queue.cc:237] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.463120 27733 raft_consensus.cc:695] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7bf97ab6f8494cdd8eb04ef0988476a8, State: Running, Role: LEADER
I20250905 19:58:18.467325 27400 catalog_manager.cc:5582] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15a3df6778944e9c8d1b663004a83903 (127.26.58.131). New cstate: current_term: 1 leader_uuid: "15a3df6778944e9c8d1b663004a83903" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } health_report { overall_health: HEALTHY } } }
I20250905 19:58:18.467247 27733 consensus_queue.cc:237] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.470227 27696 tablet_bootstrap.cc:654] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:18.475003 27401 catalog_manager.cc:5582] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:18.485669 27655 raft_consensus.cc:2466] T 7fa1390a6ad647498e7b9459277a60ac P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bf97ab6f8494cdd8eb04ef0988476a8 in term 1.
I20250905 19:58:18.489334 27697 tablet_bootstrap.cc:492] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 15a3df6778944e9c8d1b663004a83903: No bootstrap required, opened a new log
I20250905 19:58:18.489718 27696 tablet_bootstrap.cc:492] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f: No bootstrap required, opened a new log
I20250905 19:58:18.489826 27697 ts_tablet_manager.cc:1397] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 15a3df6778944e9c8d1b663004a83903: Time spent bootstrapping tablet: real 0.053s user 0.013s sys 0.029s
I20250905 19:58:18.490233 27696 ts_tablet_manager.cc:1397] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f: Time spent bootstrapping tablet: real 0.041s user 0.012s sys 0.000s
I20250905 19:58:18.490296 27717 raft_consensus.cc:491] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:18.489729 27400 catalog_manager.cc:5582] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:18.490947 27717 raft_consensus.cc:513] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.492937 27717 leader_election.cc:290] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929), 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:18.492888 27697 raft_consensus.cc:357] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.492983 27696 raft_consensus.cc:357] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.493745 27697 raft_consensus.cc:383] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:18.493774 27696 raft_consensus.cc:383] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:18.494129 27697 raft_consensus.cc:738] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15a3df6778944e9c8d1b663004a83903, State: Initialized, Role: FOLLOWER
I20250905 19:58:18.495065 27581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3ad0f154c598458fa48d8fd40ce46171" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" is_pre_election: true
I20250905 19:58:18.495034 27655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3ad0f154c598458fa48d8fd40ce46171" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15a3df6778944e9c8d1b663004a83903" is_pre_election: true
I20250905 19:58:18.494944 27697 consensus_queue.cc:260] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 15a3df6778944e9c8d1b663004a83903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.494155 27696 raft_consensus.cc:738] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Initialized, Role: FOLLOWER
I20250905 19:58:18.495757 27581 raft_consensus.cc:2466] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 025319773a4247b99829aa83dd002b0f in term 0.
I20250905 19:58:18.496901 27472 leader_election.cc:304] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025319773a4247b99829aa83dd002b0f, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters:
I20250905 19:58:18.497087 27733 raft_consensus.cc:491] T 25d5dc671025465b90a24a0f20dae08f P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:18.497539 27733 raft_consensus.cc:513] T 25d5dc671025465b90a24a0f20dae08f P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.498494 27717 raft_consensus.cc:2802] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250905 19:58:18.498838 27471 leader_election.cc:343] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:18.499559 27733 leader_election.cc:290] T 25d5dc671025465b90a24a0f20dae08f P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725), 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:18.500298 27506 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d5dc671025465b90a24a0f20dae08f" candidate_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "025319773a4247b99829aa83dd002b0f" is_pre_election: true
I20250905 19:58:18.501065 27655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25d5dc671025465b90a24a0f20dae08f" candidate_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15a3df6778944e9c8d1b663004a83903" is_pre_election: true
I20250905 19:58:18.501560 27697 ts_tablet_manager.cc:1428] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 15a3df6778944e9c8d1b663004a83903: Time spent starting tablet: real 0.011s user 0.004s sys 0.004s
I20250905 19:58:18.501803 27655 raft_consensus.cc:2391] T 25d5dc671025465b90a24a0f20dae08f P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7bf97ab6f8494cdd8eb04ef0988476a8 in current term 1: Already voted for candidate 025319773a4247b99829aa83dd002b0f in this term.
I20250905 19:58:18.502559 27697 tablet_bootstrap.cc:492] T 3ad0f154c598458fa48d8fd40ce46171 P 15a3df6778944e9c8d1b663004a83903: Bootstrap starting.
I20250905 19:58:18.498920 27717 raft_consensus.cc:491] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:18.502863 27546 leader_election.cc:304] T 25d5dc671025465b90a24a0f20dae08f P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7bf97ab6f8494cdd8eb04ef0988476a8; no voters: 025319773a4247b99829aa83dd002b0f, 15a3df6778944e9c8d1b663004a83903
I20250905 19:58:18.503168 27717 raft_consensus.cc:3058] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.504187 27733 raft_consensus.cc:3058] T 25d5dc671025465b90a24a0f20dae08f P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.510344 27697 tablet_bootstrap.cc:654] T 3ad0f154c598458fa48d8fd40ce46171 P 15a3df6778944e9c8d1b663004a83903: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:18.511111 27733 raft_consensus.cc:2747] T 25d5dc671025465b90a24a0f20dae08f P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:18.511116 27717 raft_consensus.cc:513] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.513022 27696 consensus_queue.cc:260] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.513952 27581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3ad0f154c598458fa48d8fd40ce46171" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
I20250905 19:58:18.514568 27581 raft_consensus.cc:3058] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.515244 27717 leader_election.cc:290] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 election: Requested vote from peers 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929), 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:18.518989 27655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3ad0f154c598458fa48d8fd40ce46171" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15a3df6778944e9c8d1b663004a83903"
I20250905 19:58:18.520306 27581 raft_consensus.cc:2466] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 025319773a4247b99829aa83dd002b0f in term 1.
W20250905 19:58:18.520522 27471 leader_election.cc:343] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:18.519105 27696 ts_tablet_manager.cc:1428] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20250905 19:58:18.521343 27697 tablet_bootstrap.cc:492] T 3ad0f154c598458fa48d8fd40ce46171 P 15a3df6778944e9c8d1b663004a83903: No bootstrap required, opened a new log
I20250905 19:58:18.521422 27472 leader_election.cc:304] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 025319773a4247b99829aa83dd002b0f, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters: 15a3df6778944e9c8d1b663004a83903
I20250905 19:58:18.521895 27697 ts_tablet_manager.cc:1397] T 3ad0f154c598458fa48d8fd40ce46171 P 15a3df6778944e9c8d1b663004a83903: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.004s
I20250905 19:58:18.522452 27696 tablet_bootstrap.cc:492] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f: Bootstrap starting.
I20250905 19:58:18.522603 27717 raft_consensus.cc:2802] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:18.523208 27717 raft_consensus.cc:695] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [term 1 LEADER]: Becoming Leader. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Running, Role: LEADER
I20250905 19:58:18.524262 27697 raft_consensus.cc:357] T 3ad0f154c598458fa48d8fd40ce46171 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.524742 27697 raft_consensus.cc:383] T 3ad0f154c598458fa48d8fd40ce46171 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:18.524951 27697 raft_consensus.cc:738] T 3ad0f154c598458fa48d8fd40ce46171 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15a3df6778944e9c8d1b663004a83903, State: Initialized, Role: FOLLOWER
I20250905 19:58:18.525434 27697 consensus_queue.cc:260] T 3ad0f154c598458fa48d8fd40ce46171 P 15a3df6778944e9c8d1b663004a83903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.524008 27717 consensus_queue.cc:237] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.527195 27697 ts_tablet_manager.cc:1428] T 3ad0f154c598458fa48d8fd40ce46171 P 15a3df6778944e9c8d1b663004a83903: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20250905 19:58:18.528133 27697 tablet_bootstrap.cc:492] T 650634b3e04147809fdde8cf72063c78 P 15a3df6778944e9c8d1b663004a83903: Bootstrap starting.
I20250905 19:58:18.528242 27696 tablet_bootstrap.cc:654] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:18.535805 27697 tablet_bootstrap.cc:654] T 650634b3e04147809fdde8cf72063c78 P 15a3df6778944e9c8d1b663004a83903: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:18.542637 27704 raft_consensus.cc:491] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:18.543345 27704 raft_consensus.cc:513] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.544838 27697 tablet_bootstrap.cc:492] T 650634b3e04147809fdde8cf72063c78 P 15a3df6778944e9c8d1b663004a83903: No bootstrap required, opened a new log
I20250905 19:58:18.545321 27697 ts_tablet_manager.cc:1397] T 650634b3e04147809fdde8cf72063c78 P 15a3df6778944e9c8d1b663004a83903: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20250905 19:58:18.546833 27506 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02a7a3e0f7a74cc2a1d2762c05eca777" candidate_uuid: "15a3df6778944e9c8d1b663004a83903" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "025319773a4247b99829aa83dd002b0f" is_pre_election: true
I20250905 19:58:18.547602 27506 raft_consensus.cc:2391] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 025319773a4247b99829aa83dd002b0f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 15a3df6778944e9c8d1b663004a83903 in current term 1: Already voted for candidate 7bf97ab6f8494cdd8eb04ef0988476a8 in this term.
I20250905 19:58:18.547547 27697 raft_consensus.cc:357] T 650634b3e04147809fdde8cf72063c78 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.548262 27697 raft_consensus.cc:383] T 650634b3e04147809fdde8cf72063c78 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:18.548242 27704 leader_election.cc:290] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 15a3df6778944e9c8d1b663004a83903 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725), 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929)
I20250905 19:58:18.548801 27697 raft_consensus.cc:738] T 650634b3e04147809fdde8cf72063c78 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15a3df6778944e9c8d1b663004a83903, State: Initialized, Role: FOLLOWER
I20250905 19:58:18.549669 27581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02a7a3e0f7a74cc2a1d2762c05eca777" candidate_uuid: "15a3df6778944e9c8d1b663004a83903" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" is_pre_election: true
I20250905 19:58:18.549589 27697 consensus_queue.cc:260] T 650634b3e04147809fdde8cf72063c78 P 15a3df6778944e9c8d1b663004a83903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.550711 27400 catalog_manager.cc:5582] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 025319773a4247b99829aa83dd002b0f (127.26.58.129). New cstate: current_term: 1 leader_uuid: "025319773a4247b99829aa83dd002b0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:18.553491 27697 ts_tablet_manager.cc:1428] T 650634b3e04147809fdde8cf72063c78 P 15a3df6778944e9c8d1b663004a83903: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250905 19:58:18.553754 27621 leader_election.cc:304] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 15a3df6778944e9c8d1b663004a83903 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15a3df6778944e9c8d1b663004a83903; no voters: 025319773a4247b99829aa83dd002b0f, 7bf97ab6f8494cdd8eb04ef0988476a8
I20250905 19:58:18.554410 27704 raft_consensus.cc:3058] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.558382 27696 tablet_bootstrap.cc:492] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f: No bootstrap required, opened a new log
I20250905 19:58:18.558842 27696 ts_tablet_manager.cc:1397] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f: Time spent bootstrapping tablet: real 0.037s user 0.024s sys 0.011s
I20250905 19:58:18.559753 27704 raft_consensus.cc:2747] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:18.561177 27696 raft_consensus.cc:357] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } }
I20250905 19:58:18.561771 27696 raft_consensus.cc:383] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:18.562007 27696 raft_consensus.cc:738] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Initialized, Role: FOLLOWER
I20250905 19:58:18.562572 27696 consensus_queue.cc:260] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } }
I20250905 19:58:18.564276 27696 ts_tablet_manager.cc:1428] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250905 19:58:18.565498 27717 raft_consensus.cc:491] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:18.565958 27717 raft_consensus.cc:513] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } }
I20250905 19:58:18.567929 27717 leader_election.cc:290] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617), 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929)
I20250905 19:58:18.571272 27581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a6727b25590409b9402c5aee52a55a7" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" is_pre_election: true
I20250905 19:58:18.571676 27655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a6727b25590409b9402c5aee52a55a7" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15a3df6778944e9c8d1b663004a83903" is_pre_election: true
I20250905 19:58:18.571929 27581 raft_consensus.cc:2466] T 1a6727b25590409b9402c5aee52a55a7 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 025319773a4247b99829aa83dd002b0f in term 0.
I20250905 19:58:18.572263 27655 raft_consensus.cc:2466] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 025319773a4247b99829aa83dd002b0f in term 0.
I20250905 19:58:18.573141 27472 leader_election.cc:304] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025319773a4247b99829aa83dd002b0f, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters:
I20250905 19:58:18.573915 27717 raft_consensus.cc:2802] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:18.574247 27717 raft_consensus.cc:491] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:18.574540 27717 raft_consensus.cc:3058] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.579025 27717 raft_consensus.cc:513] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } }
I20250905 19:58:18.580446 27717 leader_election.cc:290] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 election: Requested vote from peers 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617), 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929)
I20250905 19:58:18.581218 27655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a6727b25590409b9402c5aee52a55a7" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15a3df6778944e9c8d1b663004a83903"
I20250905 19:58:18.581422 27581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a6727b25590409b9402c5aee52a55a7" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
I20250905 19:58:18.581715 27655 raft_consensus.cc:3058] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.581918 27581 raft_consensus.cc:3058] T 1a6727b25590409b9402c5aee52a55a7 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.584942 27717 raft_consensus.cc:491] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:18.585403 27717 raft_consensus.cc:513] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.586521 27581 raft_consensus.cc:2466] T 1a6727b25590409b9402c5aee52a55a7 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 025319773a4247b99829aa83dd002b0f in term 1.
I20250905 19:58:18.587049 27655 raft_consensus.cc:2466] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 025319773a4247b99829aa83dd002b0f in term 1.
I20250905 19:58:18.587365 27717 leader_election.cc:290] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929), 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:18.587615 27472 leader_election.cc:304] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025319773a4247b99829aa83dd002b0f, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters:
I20250905 19:58:18.588064 27581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eb30fb918af423aba649dba5b74f749" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" is_pre_election: true
I20250905 19:58:18.588491 27717 raft_consensus.cc:2802] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:18.588833 27655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eb30fb918af423aba649dba5b74f749" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15a3df6778944e9c8d1b663004a83903" is_pre_election: true
I20250905 19:58:18.588965 27717 raft_consensus.cc:695] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [term 1 LEADER]: Becoming Leader. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Running, Role: LEADER
I20250905 19:58:18.589556 27655 raft_consensus.cc:2391] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 025319773a4247b99829aa83dd002b0f in current term 1: Already voted for candidate 7bf97ab6f8494cdd8eb04ef0988476a8 in this term.
I20250905 19:58:18.589766 27717 consensus_queue.cc:237] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } }
I20250905 19:58:18.590334 27471 leader_election.cc:304] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 025319773a4247b99829aa83dd002b0f; no voters: 15a3df6778944e9c8d1b663004a83903, 7bf97ab6f8494cdd8eb04ef0988476a8
I20250905 19:58:18.590975 27725 raft_consensus.cc:3058] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.597673 27725 raft_consensus.cc:2747] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:18.597162 27400 catalog_manager.cc:5582] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 025319773a4247b99829aa83dd002b0f (127.26.58.129). New cstate: current_term: 1 leader_uuid: "025319773a4247b99829aa83dd002b0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } health_report { overall_health: HEALTHY } } }
I20250905 19:58:18.603592 27725 raft_consensus.cc:491] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:18.604043 27725 raft_consensus.cc:513] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.605777 27725 leader_election.cc:290] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929), 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:18.606555 27581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccec5ef5fb0d4383a93e6cf0514aefb8" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" is_pre_election: true
I20250905 19:58:18.606812 27655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccec5ef5fb0d4383a93e6cf0514aefb8" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15a3df6778944e9c8d1b663004a83903" is_pre_election: true
I20250905 19:58:18.607249 27581 raft_consensus.cc:2391] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 025319773a4247b99829aa83dd002b0f in current term 1: Already voted for candidate 15a3df6778944e9c8d1b663004a83903 in this term.
I20250905 19:58:18.608428 27472 leader_election.cc:304] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 025319773a4247b99829aa83dd002b0f; no voters: 15a3df6778944e9c8d1b663004a83903, 7bf97ab6f8494cdd8eb04ef0988476a8
I20250905 19:58:18.609045 27725 raft_consensus.cc:3058] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.613965 27725 raft_consensus.cc:2747] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:18.642416 27725 consensus_queue.cc:1035] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:18.652756 27702 consensus_queue.cc:1035] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:18.699530 27655 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "fa10aea786b5462c82a92fc0f9bff996"
dest_uuid: "15a3df6778944e9c8d1b663004a83903"
mode: GRACEFUL
new_leader_uuid: "025319773a4247b99829aa83dd002b0f"
from {username='slave'} at 127.0.0.1:55086
I20250905 19:58:18.700124 27655 raft_consensus.cc:604] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [term 1 LEADER]: Received request to transfer leadership to TS 025319773a4247b99829aa83dd002b0f
I20250905 19:58:18.702606 27733 consensus_queue.cc:1035] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:18.714459 27700 consensus_queue.cc:1035] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:18.720010 27581 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5eb30fb918af423aba649dba5b74f749"
dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
mode: GRACEFUL
new_leader_uuid: "025319773a4247b99829aa83dd002b0f"
from {username='slave'} at 127.0.0.1:52276
I20250905 19:58:18.720598 27581 raft_consensus.cc:604] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 LEADER]: Received request to transfer leadership to TS 025319773a4247b99829aa83dd002b0f
I20250905 19:58:18.731213 27655 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "ccec5ef5fb0d4383a93e6cf0514aefb8"
dest_uuid: "15a3df6778944e9c8d1b663004a83903"
mode: GRACEFUL
new_leader_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
from {username='slave'} at 127.0.0.1:55086
I20250905 19:58:18.731855 27655 raft_consensus.cc:604] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [term 1 LEADER]: Received request to transfer leadership to TS 7bf97ab6f8494cdd8eb04ef0988476a8
I20250905 19:58:18.744361 27702 consensus_queue.cc:1035] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:18.750964 27506 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "3ad0f154c598458fa48d8fd40ce46171"
dest_uuid: "025319773a4247b99829aa83dd002b0f"
mode: GRACEFUL
new_leader_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
from {username='slave'} at 127.0.0.1:40192
I20250905 19:58:18.751586 27506 raft_consensus.cc:604] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [term 1 LEADER]: Received request to transfer leadership to TS 7bf97ab6f8494cdd8eb04ef0988476a8
I20250905 19:58:18.757169 27702 consensus_queue.cc:1035] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:18.757684 27506 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "650634b3e04147809fdde8cf72063c78"
dest_uuid: "025319773a4247b99829aa83dd002b0f"
mode: GRACEFUL
new_leader_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
from {username='slave'} at 127.0.0.1:40192
I20250905 19:58:18.758246 27506 raft_consensus.cc:604] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [term 1 LEADER]: Received request to transfer leadership to TS 7bf97ab6f8494cdd8eb04ef0988476a8
I20250905 19:58:18.764083 27581 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "02a7a3e0f7a74cc2a1d2762c05eca777"
dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
mode: GRACEFUL
new_leader_uuid: "15a3df6778944e9c8d1b663004a83903"
from {username='slave'} at 127.0.0.1:52276
I20250905 19:58:18.764570 27581 raft_consensus.cc:604] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 LEADER]: Received request to transfer leadership to TS 15a3df6778944e9c8d1b663004a83903
I20250905 19:58:18.774420 27704 consensus_queue.cc:1035] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [LEADER]: Connected to new peer: Peer: permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:18.780336 27738 raft_consensus.cc:991] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903: : Instructing follower 025319773a4247b99829aa83dd002b0f to start an election
I20250905 19:58:18.780949 27746 raft_consensus.cc:1079] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [term 1 LEADER]: Signalling peer 025319773a4247b99829aa83dd002b0f to start an election
I20250905 19:58:18.783388 27505 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "fa10aea786b5462c82a92fc0f9bff996"
dest_uuid: "025319773a4247b99829aa83dd002b0f"
from {username='slave'} at 127.0.0.1:40178
I20250905 19:58:18.784628 27505 raft_consensus.cc:491] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:58:18.785012 27505 raft_consensus.cc:3058] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:18.786255 27746 consensus_queue.cc:1035] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:18.791893 27505 raft_consensus.cc:513] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.794200 27505 leader_election.cc:290] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 2 election: Requested vote from peers 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929), 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:18.795010 27581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa10aea786b5462c82a92fc0f9bff996" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
I20250905 19:58:18.795663 27581 raft_consensus.cc:3058] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:18.797437 27655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa10aea786b5462c82a92fc0f9bff996" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "15a3df6778944e9c8d1b663004a83903"
I20250905 19:58:18.798035 27655 raft_consensus.cc:3053] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:58:18.798365 27655 raft_consensus.cc:738] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 15a3df6778944e9c8d1b663004a83903, State: Running, Role: LEADER
I20250905 19:58:18.799221 27655 consensus_queue.cc:260] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.800638 27655 raft_consensus.cc:3058] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:18.802578 27581 raft_consensus.cc:2466] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 025319773a4247b99829aa83dd002b0f in term 2.
I20250905 19:58:18.803604 27472 leader_election.cc:304] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025319773a4247b99829aa83dd002b0f, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters:
I20250905 19:58:18.804240 27702 raft_consensus.cc:2802] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:58:18.804625 27702 raft_consensus.cc:695] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f [term 2 LEADER]: Becoming Leader. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Running, Role: LEADER
I20250905 19:58:18.805433 27702 consensus_queue.cc:237] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.806993 27655 raft_consensus.cc:2466] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 025319773a4247b99829aa83dd002b0f in term 2.
I20250905 19:58:18.811334 27746 consensus_queue.cc:1035] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [LEADER]: Connected to new peer: Peer: permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:18.812543 27400 catalog_manager.cc:5582] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f reported cstate change: term changed from 1 to 2, leader changed from 15a3df6778944e9c8d1b663004a83903 (127.26.58.131) to 025319773a4247b99829aa83dd002b0f (127.26.58.129). New cstate: current_term: 2 leader_uuid: "025319773a4247b99829aa83dd002b0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:18.820860 27738 consensus_queue.cc:1035] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:18.831447 27746 raft_consensus.cc:991] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903: : Instructing follower 7bf97ab6f8494cdd8eb04ef0988476a8 to start an election
I20250905 19:58:18.831948 27738 raft_consensus.cc:1079] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [term 1 LEADER]: Signalling peer 7bf97ab6f8494cdd8eb04ef0988476a8 to start an election
I20250905 19:58:18.833557 27581 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "ccec5ef5fb0d4383a93e6cf0514aefb8"
dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
from {username='slave'} at 127.0.0.1:52266
I20250905 19:58:18.834215 27581 raft_consensus.cc:491] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:58:18.834625 27581 raft_consensus.cc:3058] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:18.839186 27581 raft_consensus.cc:513] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.840716 27581 leader_election.cc:290] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 2 election: Requested vote from peers 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725), 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:18.841463 27505 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccec5ef5fb0d4383a93e6cf0514aefb8" candidate_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "025319773a4247b99829aa83dd002b0f"
I20250905 19:58:18.841749 27655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccec5ef5fb0d4383a93e6cf0514aefb8" candidate_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "15a3df6778944e9c8d1b663004a83903"
I20250905 19:58:18.842020 27505 raft_consensus.cc:3058] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:18.842267 27655 raft_consensus.cc:3053] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:58:18.842628 27655 raft_consensus.cc:738] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 15a3df6778944e9c8d1b663004a83903, State: Running, Role: LEADER
I20250905 19:58:18.843400 27655 consensus_queue.cc:260] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.844668 27655 raft_consensus.cc:3058] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:18.848896 27505 raft_consensus.cc:2466] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bf97ab6f8494cdd8eb04ef0988476a8 in term 2.
I20250905 19:58:18.849989 27547 leader_election.cc:304] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025319773a4247b99829aa83dd002b0f, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters:
I20250905 19:58:18.850785 27733 raft_consensus.cc:2802] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:58:18.851157 27733 raft_consensus.cc:695] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 LEADER]: Becoming Leader. State: Replica: 7bf97ab6f8494cdd8eb04ef0988476a8, State: Running, Role: LEADER
I20250905 19:58:18.851341 27655 raft_consensus.cc:2466] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bf97ab6f8494cdd8eb04ef0988476a8 in term 2.
I20250905 19:58:18.851960 27733 consensus_queue.cc:237] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.855530 27655 raft_consensus.cc:3058] T 650634b3e04147809fdde8cf72063c78 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.858181 27400 catalog_manager.cc:5582] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 reported cstate change: term changed from 1 to 2, leader changed from 15a3df6778944e9c8d1b663004a83903 (127.26.58.131) to 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130). New cstate: current_term: 2 leader_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:18.861230 27702 consensus_queue.cc:1035] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:18.861524 27738 raft_consensus.cc:491] T 650634b3e04147809fdde8cf72063c78 P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:18.861958 27738 raft_consensus.cc:513] T 650634b3e04147809fdde8cf72063c78 P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.863782 27738 leader_election.cc:290] T 650634b3e04147809fdde8cf72063c78 P 15a3df6778944e9c8d1b663004a83903 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725), 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929)
I20250905 19:58:18.864807 27505 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "650634b3e04147809fdde8cf72063c78" candidate_uuid: "15a3df6778944e9c8d1b663004a83903" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "025319773a4247b99829aa83dd002b0f" is_pre_election: true
I20250905 19:58:18.866735 27581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "650634b3e04147809fdde8cf72063c78" candidate_uuid: "15a3df6778944e9c8d1b663004a83903" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" is_pre_election: true
I20250905 19:58:18.867555 27581 raft_consensus.cc:2466] T 650634b3e04147809fdde8cf72063c78 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15a3df6778944e9c8d1b663004a83903 in term 1.
I20250905 19:58:18.868605 27621 leader_election.cc:304] T 650634b3e04147809fdde8cf72063c78 P 15a3df6778944e9c8d1b663004a83903 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 15a3df6778944e9c8d1b663004a83903, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters: 025319773a4247b99829aa83dd002b0f
I20250905 19:58:18.869411 27746 raft_consensus.cc:2802] T 650634b3e04147809fdde8cf72063c78 P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:58:18.869773 27746 raft_consensus.cc:491] T 650634b3e04147809fdde8cf72063c78 P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 025319773a4247b99829aa83dd002b0f)
I20250905 19:58:18.870152 27746 raft_consensus.cc:3058] T 650634b3e04147809fdde8cf72063c78 P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:18.875031 27725 consensus_queue.cc:1035] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:18.877923 27746 raft_consensus.cc:513] T 650634b3e04147809fdde8cf72063c78 P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.880347 27655 raft_consensus.cc:1238] T 650634b3e04147809fdde8cf72063c78 P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Rejecting Update request from peer 025319773a4247b99829aa83dd002b0f for earlier term 1. Current term is 2. Ops: []
I20250905 19:58:18.880854 27505 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "650634b3e04147809fdde8cf72063c78" candidate_uuid: "15a3df6778944e9c8d1b663004a83903" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "025319773a4247b99829aa83dd002b0f"
I20250905 19:58:18.883082 27746 leader_election.cc:290] T 650634b3e04147809fdde8cf72063c78 P 15a3df6778944e9c8d1b663004a83903 [CANDIDATE]: Term 2 election: Requested vote from peers 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725), 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929)
I20250905 19:58:18.884052 27581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "650634b3e04147809fdde8cf72063c78" candidate_uuid: "15a3df6778944e9c8d1b663004a83903" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
I20250905 19:58:18.884845 27702 consensus_queue.cc:1046] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:18.885486 27717 raft_consensus.cc:991] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f: : Instructing follower 7bf97ab6f8494cdd8eb04ef0988476a8 to start an election
I20250905 19:58:18.885529 27621 leader_election.cc:304] T 650634b3e04147809fdde8cf72063c78 P 15a3df6778944e9c8d1b663004a83903 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15a3df6778944e9c8d1b663004a83903; no voters: 025319773a4247b99829aa83dd002b0f, 7bf97ab6f8494cdd8eb04ef0988476a8
I20250905 19:58:18.885897 27702 raft_consensus.cc:1079] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [term 1 LEADER]: Signalling peer 7bf97ab6f8494cdd8eb04ef0988476a8 to start an election
I20250905 19:58:18.886366 27746 raft_consensus.cc:2747] T 650634b3e04147809fdde8cf72063c78 P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 19:58:18.886811 27717 raft_consensus.cc:3053] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:58:18.887158 27717 raft_consensus.cc:738] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Running, Role: LEADER
I20250905 19:58:18.887537 27581 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "650634b3e04147809fdde8cf72063c78"
dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
from {username='slave'} at 127.0.0.1:52260
I20250905 19:58:18.888042 27581 raft_consensus.cc:491] T 650634b3e04147809fdde8cf72063c78 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:58:18.888412 27581 raft_consensus.cc:3058] T 650634b3e04147809fdde8cf72063c78 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:18.887902 27717 consensus_queue.cc:260] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.889130 27717 raft_consensus.cc:3058] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:18.895304 27581 raft_consensus.cc:513] T 650634b3e04147809fdde8cf72063c78 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.897038 27581 leader_election.cc:290] T 650634b3e04147809fdde8cf72063c78 P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 2 election: Requested vote from peers 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725), 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:18.897980 27505 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "650634b3e04147809fdde8cf72063c78" candidate_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "025319773a4247b99829aa83dd002b0f"
I20250905 19:58:18.898310 27655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "650634b3e04147809fdde8cf72063c78" candidate_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "15a3df6778944e9c8d1b663004a83903"
I20250905 19:58:18.899205 27655 raft_consensus.cc:2391] T 650634b3e04147809fdde8cf72063c78 P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7bf97ab6f8494cdd8eb04ef0988476a8 in current term 2: Already voted for candidate 15a3df6778944e9c8d1b663004a83903 in this term.
I20250905 19:58:18.904395 27505 raft_consensus.cc:2466] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bf97ab6f8494cdd8eb04ef0988476a8 in term 2.
I20250905 19:58:18.905241 27547 leader_election.cc:304] T 650634b3e04147809fdde8cf72063c78 P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 025319773a4247b99829aa83dd002b0f, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters: 15a3df6778944e9c8d1b663004a83903
I20250905 19:58:18.905876 27733 raft_consensus.cc:2802] T 650634b3e04147809fdde8cf72063c78 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:58:18.906252 27733 raft_consensus.cc:695] T 650634b3e04147809fdde8cf72063c78 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 LEADER]: Becoming Leader. State: Replica: 7bf97ab6f8494cdd8eb04ef0988476a8, State: Running, Role: LEADER
I20250905 19:58:18.907011 27733 consensus_queue.cc:237] T 650634b3e04147809fdde8cf72063c78 P 7bf97ab6f8494cdd8eb04ef0988476a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.913620 27400 catalog_manager.cc:5582] T 650634b3e04147809fdde8cf72063c78 P 7bf97ab6f8494cdd8eb04ef0988476a8 reported cstate change: term changed from 1 to 2, leader changed from 025319773a4247b99829aa83dd002b0f (127.26.58.129) to 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130). New cstate: current_term: 2 leader_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:18.927466 27746 raft_consensus.cc:491] T 3ad0f154c598458fa48d8fd40ce46171 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:18.927882 27746 raft_consensus.cc:513] T 3ad0f154c598458fa48d8fd40ce46171 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.929304 27746 leader_election.cc:290] T 3ad0f154c598458fa48d8fd40ce46171 P 15a3df6778944e9c8d1b663004a83903 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725), 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929)
I20250905 19:58:18.930071 27505 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3ad0f154c598458fa48d8fd40ce46171" candidate_uuid: "15a3df6778944e9c8d1b663004a83903" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "025319773a4247b99829aa83dd002b0f" is_pre_election: true
I20250905 19:58:18.930209 27581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3ad0f154c598458fa48d8fd40ce46171" candidate_uuid: "15a3df6778944e9c8d1b663004a83903" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" is_pre_election: true
I20250905 19:58:18.930895 27581 raft_consensus.cc:2391] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 15a3df6778944e9c8d1b663004a83903 in current term 1: Already voted for candidate 025319773a4247b99829aa83dd002b0f in this term.
I20250905 19:58:18.931748 27621 leader_election.cc:304] T 3ad0f154c598458fa48d8fd40ce46171 P 15a3df6778944e9c8d1b663004a83903 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15a3df6778944e9c8d1b663004a83903; no voters: 025319773a4247b99829aa83dd002b0f, 7bf97ab6f8494cdd8eb04ef0988476a8
I20250905 19:58:18.932305 27746 raft_consensus.cc:3058] T 3ad0f154c598458fa48d8fd40ce46171 P 15a3df6778944e9c8d1b663004a83903 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:18.936836 27746 raft_consensus.cc:2747] T 3ad0f154c598458fa48d8fd40ce46171 P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:18.940245 27733 consensus_queue.cc:1035] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:18.942885 27733 consensus_queue.cc:1035] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:18.954824 27729 consensus_queue.cc:1035] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:18.962993 27778 consensus_queue.cc:1035] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:18.972239 27778 raft_consensus.cc:991] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8: : Instructing follower 025319773a4247b99829aa83dd002b0f to start an election
I20250905 19:58:18.972707 27733 raft_consensus.cc:1079] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 LEADER]: Signalling peer 025319773a4247b99829aa83dd002b0f to start an election
I20250905 19:58:18.974381 27505 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "5eb30fb918af423aba649dba5b74f749"
dest_uuid: "025319773a4247b99829aa83dd002b0f"
from {username='slave'} at 127.0.0.1:40184
I20250905 19:58:18.974980 27505 raft_consensus.cc:491] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:58:18.975356 27505 raft_consensus.cc:3058] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:18.982097 27505 raft_consensus.cc:513] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.984380 27505 leader_election.cc:290] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 2 election: Requested vote from peers 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929), 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:18.984632 27654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eb30fb918af423aba649dba5b74f749" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "15a3df6778944e9c8d1b663004a83903"
I20250905 19:58:18.985208 27654 raft_consensus.cc:3058] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:18.985890 27581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eb30fb918af423aba649dba5b74f749" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
I20250905 19:58:18.986317 27581 raft_consensus.cc:3053] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:58:18.986639 27581 raft_consensus.cc:738] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7bf97ab6f8494cdd8eb04ef0988476a8, State: Running, Role: LEADER
I20250905 19:58:18.987376 27581 consensus_queue.cc:260] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.988663 27581 raft_consensus.cc:3058] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:18.990689 27654 raft_consensus.cc:2466] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 025319773a4247b99829aa83dd002b0f in term 2.
I20250905 19:58:18.991585 27471 leader_election.cc:304] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025319773a4247b99829aa83dd002b0f, 15a3df6778944e9c8d1b663004a83903; no voters:
I20250905 19:58:18.992352 27717 raft_consensus.cc:2802] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:58:18.992749 27717 raft_consensus.cc:695] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [term 2 LEADER]: Becoming Leader. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Running, Role: LEADER
I20250905 19:58:18.993476 27717 consensus_queue.cc:237] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:18.994105 27581 raft_consensus.cc:2466] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 025319773a4247b99829aa83dd002b0f in term 2.
I20250905 19:58:18.999605 27400 catalog_manager.cc:5582] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f reported cstate change: term changed from 1 to 2, leader changed from 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130) to 025319773a4247b99829aa83dd002b0f (127.26.58.129). New cstate: current_term: 2 leader_uuid: "025319773a4247b99829aa83dd002b0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:19.056679 27717 consensus_queue.cc:1035] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:19.060432 27702 consensus_queue.cc:1035] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:19.071131 27725 consensus_queue.cc:1035] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:19.072067 27783 raft_consensus.cc:991] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f: : Instructing follower 7bf97ab6f8494cdd8eb04ef0988476a8 to start an election
I20250905 19:58:19.072644 27717 raft_consensus.cc:1079] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [term 1 LEADER]: Signalling peer 7bf97ab6f8494cdd8eb04ef0988476a8 to start an election
I20250905 19:58:19.074527 27580 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "3ad0f154c598458fa48d8fd40ce46171"
dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
from {username='slave'} at 127.0.0.1:52260
I20250905 19:58:19.075124 27580 raft_consensus.cc:491] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:58:19.075503 27580 raft_consensus.cc:3058] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:19.086156 27580 raft_consensus.cc:513] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:19.087321 27702 consensus_queue.cc:1035] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:19.088055 27580 leader_election.cc:290] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 2 election: Requested vote from peers 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725), 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:19.088810 27505 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3ad0f154c598458fa48d8fd40ce46171" candidate_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "025319773a4247b99829aa83dd002b0f"
I20250905 19:58:19.089391 27505 raft_consensus.cc:3053] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:58:19.089629 27655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3ad0f154c598458fa48d8fd40ce46171" candidate_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "15a3df6778944e9c8d1b663004a83903"
I20250905 19:58:19.089916 27581 raft_consensus.cc:1238] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Rejecting Update request from peer 025319773a4247b99829aa83dd002b0f for earlier term 1. Current term is 2. Ops: []
I20250905 19:58:19.090262 27655 raft_consensus.cc:3058] T 3ad0f154c598458fa48d8fd40ce46171 P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:19.089725 27505 raft_consensus.cc:738] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Running, Role: LEADER
I20250905 19:58:19.091622 27505 consensus_queue.cc:260] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:19.092296 27505 raft_consensus.cc:3058] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:19.097118 27655 raft_consensus.cc:2466] T 3ad0f154c598458fa48d8fd40ce46171 P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bf97ab6f8494cdd8eb04ef0988476a8 in term 2.
I20250905 19:58:19.097868 27654 raft_consensus.cc:1238] T 3ad0f154c598458fa48d8fd40ce46171 P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Rejecting Update request from peer 025319773a4247b99829aa83dd002b0f for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20250905 19:58:19.098131 27546 leader_election.cc:304] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15a3df6778944e9c8d1b663004a83903, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters:
I20250905 19:58:19.098331 27505 raft_consensus.cc:2466] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bf97ab6f8494cdd8eb04ef0988476a8 in term 2.
I20250905 19:58:19.098976 27733 raft_consensus.cc:2802] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:58:19.099449 27733 raft_consensus.cc:695] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 LEADER]: Becoming Leader. State: Replica: 7bf97ab6f8494cdd8eb04ef0988476a8, State: Running, Role: LEADER
I20250905 19:58:19.100291 27733 consensus_queue.cc:237] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:19.106163 27404 catalog_manager.cc:5582] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 reported cstate change: term changed from 1 to 2, leader changed from 025319773a4247b99829aa83dd002b0f (127.26.58.129) to 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130). New cstate: current_term: 2 leader_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:19.259173 27654 raft_consensus.cc:1273] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Refusing update from remote peer 7bf97ab6f8494cdd8eb04ef0988476a8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:58:19.260314 27733 consensus_queue.cc:1035] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:58:19.267966 27505 raft_consensus.cc:1273] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Refusing update from remote peer 7bf97ab6f8494cdd8eb04ef0988476a8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:58:19.269160 27700 consensus_queue.cc:1035] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:58:19.339654 27733 raft_consensus.cc:991] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8: : Instructing follower 15a3df6778944e9c8d1b663004a83903 to start an election
I20250905 19:58:19.340129 27700 raft_consensus.cc:1079] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 LEADER]: Signalling peer 15a3df6778944e9c8d1b663004a83903 to start an election
I20250905 19:58:19.341533 27654 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "02a7a3e0f7a74cc2a1d2762c05eca777"
dest_uuid: "15a3df6778944e9c8d1b663004a83903"
from {username='slave'} at 127.0.0.1:55084
I20250905 19:58:19.342059 27654 raft_consensus.cc:491] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:58:19.342415 27654 raft_consensus.cc:3058] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:19.347189 27654 raft_consensus.cc:513] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:19.348575 27654 leader_election.cc:290] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 15a3df6778944e9c8d1b663004a83903 [CANDIDATE]: Term 2 election: Requested vote from peers 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725), 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929)
I20250905 19:58:19.349455 27505 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02a7a3e0f7a74cc2a1d2762c05eca777" candidate_uuid: "15a3df6778944e9c8d1b663004a83903" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "025319773a4247b99829aa83dd002b0f"
I20250905 19:58:19.349978 27581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02a7a3e0f7a74cc2a1d2762c05eca777" candidate_uuid: "15a3df6778944e9c8d1b663004a83903" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
I20250905 19:58:19.350474 27505 raft_consensus.cc:3058] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 025319773a4247b99829aa83dd002b0f [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:19.350517 27581 raft_consensus.cc:3053] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:58:19.351004 27581 raft_consensus.cc:738] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7bf97ab6f8494cdd8eb04ef0988476a8, State: Running, Role: LEADER
I20250905 19:58:19.351800 27581 consensus_queue.cc:260] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:19.353008 27581 raft_consensus.cc:3058] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:19.357685 27505 raft_consensus.cc:2466] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15a3df6778944e9c8d1b663004a83903 in term 2.
I20250905 19:58:19.358870 27621 leader_election.cc:304] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 15a3df6778944e9c8d1b663004a83903 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025319773a4247b99829aa83dd002b0f, 15a3df6778944e9c8d1b663004a83903; no voters:
I20250905 19:58:19.359427 27581 raft_consensus.cc:2466] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15a3df6778944e9c8d1b663004a83903 in term 2.
I20250905 19:58:19.359531 27746 raft_consensus.cc:2802] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:58:19.359973 27746 raft_consensus.cc:695] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 15a3df6778944e9c8d1b663004a83903 [term 2 LEADER]: Becoming Leader. State: Replica: 15a3df6778944e9c8d1b663004a83903, State: Running, Role: LEADER
I20250905 19:58:19.360667 27746 consensus_queue.cc:237] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 15a3df6778944e9c8d1b663004a83903 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:19.366477 27404 catalog_manager.cc:5582] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 15a3df6778944e9c8d1b663004a83903 reported cstate change: term changed from 1 to 2, leader changed from 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130) to 15a3df6778944e9c8d1b663004a83903 (127.26.58.131). New cstate: current_term: 2 leader_uuid: "15a3df6778944e9c8d1b663004a83903" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } health_report { overall_health: HEALTHY } } }
I20250905 19:58:19.372215 27581 raft_consensus.cc:1273] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Refusing update from remote peer 025319773a4247b99829aa83dd002b0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:58:19.373596 27702 consensus_queue.cc:1035] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:58:19.377718 27580 raft_consensus.cc:1273] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Refusing update from remote peer 025319773a4247b99829aa83dd002b0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:58:19.379084 27702 consensus_queue.cc:1035] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:58:19.381929 27654 raft_consensus.cc:1273] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Refusing update from remote peer 025319773a4247b99829aa83dd002b0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:58:19.383404 27702 consensus_queue.cc:1035] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:58:19.388295 27655 raft_consensus.cc:1273] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Refusing update from remote peer 025319773a4247b99829aa83dd002b0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:58:19.389914 27725 consensus_queue.cc:1035] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:58:19.453653 27505 raft_consensus.cc:1273] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Refusing update from remote peer 7bf97ab6f8494cdd8eb04ef0988476a8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:58:19.454885 27700 consensus_queue.cc:1035] T 650634b3e04147809fdde8cf72063c78 P 7bf97ab6f8494cdd8eb04ef0988476a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:58:19.462204 27654 raft_consensus.cc:1273] T 650634b3e04147809fdde8cf72063c78 P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Refusing update from remote peer 7bf97ab6f8494cdd8eb04ef0988476a8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:58:19.463752 27700 consensus_queue.cc:1035] T 650634b3e04147809fdde8cf72063c78 P 7bf97ab6f8494cdd8eb04ef0988476a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:19.591485 27654 raft_consensus.cc:1273] T 3ad0f154c598458fa48d8fd40ce46171 P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Refusing update from remote peer 7bf97ab6f8494cdd8eb04ef0988476a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:58:19.592674 27700 consensus_queue.cc:1035] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:19.602025 27505 raft_consensus.cc:1273] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Refusing update from remote peer 7bf97ab6f8494cdd8eb04ef0988476a8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:58:19.603518 27778 consensus_queue.cc:1035] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:58:19.739929 27580 raft_consensus.cc:1273] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Refusing update from remote peer 15a3df6778944e9c8d1b663004a83903: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:58:19.741209 27746 consensus_queue.cc:1035] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 15a3df6778944e9c8d1b663004a83903 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:58:19.749933 27505 raft_consensus.cc:1273] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Refusing update from remote peer 15a3df6778944e9c8d1b663004a83903: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:58:19.751796 27788 consensus_queue.cc:1035] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 15a3df6778944e9c8d1b663004a83903 [LEADER]: Connected to new peer: Peer: permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:58:21.793975 26858 auto_leader_rebalancer-test.cc:343] The leader distribution is
I20250905 19:58:21.794596 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:21.811337 27506 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "1a6727b25590409b9402c5aee52a55a7"
dest_uuid: "025319773a4247b99829aa83dd002b0f"
mode: GRACEFUL
new_leader_uuid: "15a3df6778944e9c8d1b663004a83903"
from {username='slave'} at 127.0.0.1:40198
I20250905 19:58:21.811784 27506 raft_consensus.cc:604] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [term 1 LEADER]: Received request to transfer leadership to TS 15a3df6778944e9c8d1b663004a83903
I20250905 19:58:21.813723 27506 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "66b1b69c203145e4bcc2cb26522c663f"
dest_uuid: "025319773a4247b99829aa83dd002b0f"
mode: GRACEFUL
new_leader_uuid: "15a3df6778944e9c8d1b663004a83903"
from {username='slave'} at 127.0.0.1:40198
I20250905 19:58:21.814122 27506 raft_consensus.cc:604] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [term 1 LEADER]: Received request to transfer leadership to TS 15a3df6778944e9c8d1b663004a83903
I20250905 19:58:21.814960 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250905 19:58:21.815379 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:22.259264 27792 raft_consensus.cc:991] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f: : Instructing follower 15a3df6778944e9c8d1b663004a83903 to start an election
I20250905 19:58:22.259645 27795 raft_consensus.cc:1079] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [term 1 LEADER]: Signalling peer 15a3df6778944e9c8d1b663004a83903 to start an election
I20250905 19:58:22.260938 27654 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "1a6727b25590409b9402c5aee52a55a7"
dest_uuid: "15a3df6778944e9c8d1b663004a83903"
from {username='slave'} at 127.0.0.1:55072
I20250905 19:58:22.261385 27654 raft_consensus.cc:491] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:58:22.261726 27654 raft_consensus.cc:3058] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:22.268994 27654 raft_consensus.cc:513] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } }
I20250905 19:58:22.271099 27654 leader_election.cc:290] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903 [CANDIDATE]: Term 2 election: Requested vote from peers 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929), 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725)
I20250905 19:58:22.272598 27506 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a6727b25590409b9402c5aee52a55a7" candidate_uuid: "15a3df6778944e9c8d1b663004a83903" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "025319773a4247b99829aa83dd002b0f"
I20250905 19:58:22.272859 27580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a6727b25590409b9402c5aee52a55a7" candidate_uuid: "15a3df6778944e9c8d1b663004a83903" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
I20250905 19:58:22.273121 27506 raft_consensus.cc:3053] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:58:22.273379 27580 raft_consensus.cc:3058] T 1a6727b25590409b9402c5aee52a55a7 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:22.273490 27506 raft_consensus.cc:738] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Running, Role: LEADER
I20250905 19:58:22.274325 27506 consensus_queue.cc:260] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } }
I20250905 19:58:22.275655 27506 raft_consensus.cc:3058] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:22.278247 27580 raft_consensus.cc:2466] T 1a6727b25590409b9402c5aee52a55a7 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15a3df6778944e9c8d1b663004a83903 in term 2.
I20250905 19:58:22.279297 27621 leader_election.cc:304] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15a3df6778944e9c8d1b663004a83903, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters:
I20250905 19:58:22.279924 27738 raft_consensus.cc:2802] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:58:22.280820 27738 raft_consensus.cc:695] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903 [term 2 LEADER]: Becoming Leader. State: Replica: 15a3df6778944e9c8d1b663004a83903, State: Running, Role: LEADER
I20250905 19:58:22.281613 27506 raft_consensus.cc:2466] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15a3df6778944e9c8d1b663004a83903 in term 2.
I20250905 19:58:22.281639 27738 consensus_queue.cc:237] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } }
I20250905 19:58:22.288524 27401 catalog_manager.cc:5582] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903 reported cstate change: term changed from 1 to 2, leader changed from 025319773a4247b99829aa83dd002b0f (127.26.58.129) to 15a3df6778944e9c8d1b663004a83903 (127.26.58.131). New cstate: current_term: 2 leader_uuid: "15a3df6778944e9c8d1b663004a83903" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:22.348387 27792 raft_consensus.cc:991] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f: : Instructing follower 15a3df6778944e9c8d1b663004a83903 to start an election
I20250905 19:58:22.348739 27795 raft_consensus.cc:1079] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [term 1 LEADER]: Signalling peer 15a3df6778944e9c8d1b663004a83903 to start an election
I20250905 19:58:22.350121 27654 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "66b1b69c203145e4bcc2cb26522c663f"
dest_uuid: "15a3df6778944e9c8d1b663004a83903"
from {username='slave'} at 127.0.0.1:55072
I20250905 19:58:22.350661 27654 raft_consensus.cc:491] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:58:22.350950 27654 raft_consensus.cc:3058] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:22.355695 27654 raft_consensus.cc:513] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:22.357208 27654 leader_election.cc:290] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903 [CANDIDATE]: Term 2 election: Requested vote from peers 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725), 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929)
I20250905 19:58:22.357877 27506 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66b1b69c203145e4bcc2cb26522c663f" candidate_uuid: "15a3df6778944e9c8d1b663004a83903" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "025319773a4247b99829aa83dd002b0f"
I20250905 19:58:22.358104 27580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66b1b69c203145e4bcc2cb26522c663f" candidate_uuid: "15a3df6778944e9c8d1b663004a83903" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
I20250905 19:58:22.358481 27506 raft_consensus.cc:3053] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:58:22.358611 27580 raft_consensus.cc:3058] T 66b1b69c203145e4bcc2cb26522c663f P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:22.358768 27506 raft_consensus.cc:738] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Running, Role: LEADER
I20250905 19:58:22.359455 27506 consensus_queue.cc:260] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:22.360322 27506 raft_consensus.cc:3058] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:22.364665 27580 raft_consensus.cc:2466] T 66b1b69c203145e4bcc2cb26522c663f P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15a3df6778944e9c8d1b663004a83903 in term 2.
I20250905 19:58:22.364730 27506 raft_consensus.cc:2466] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15a3df6778944e9c8d1b663004a83903 in term 2.
I20250905 19:58:22.365742 27621 leader_election.cc:304] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15a3df6778944e9c8d1b663004a83903, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters:
I20250905 19:58:22.366398 27738 raft_consensus.cc:2802] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:58:22.366888 27738 raft_consensus.cc:695] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903 [term 2 LEADER]: Becoming Leader. State: Replica: 15a3df6778944e9c8d1b663004a83903, State: Running, Role: LEADER
I20250905 19:58:22.367578 27738 consensus_queue.cc:237] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:22.373628 27401 catalog_manager.cc:5582] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903 reported cstate change: term changed from 1 to 2, leader changed from 025319773a4247b99829aa83dd002b0f (127.26.58.129) to 15a3df6778944e9c8d1b663004a83903 (127.26.58.131). New cstate: current_term: 2 leader_uuid: "15a3df6778944e9c8d1b663004a83903" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } health_report { overall_health: HEALTHY } } }
I20250905 19:58:22.702653 27506 raft_consensus.cc:1273] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Refusing update from remote peer 15a3df6778944e9c8d1b663004a83903: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:58:22.703706 27738 consensus_queue.cc:1035] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903 [LEADER]: Connected to new peer: Peer: permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:58:22.713028 27580 raft_consensus.cc:1273] T 1a6727b25590409b9402c5aee52a55a7 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Refusing update from remote peer 15a3df6778944e9c8d1b663004a83903: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:58:22.714628 27738 consensus_queue.cc:1035] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:58:22.816309 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:22.822242 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:22.822623 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:22.955940 27506 raft_consensus.cc:1273] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Refusing update from remote peer 15a3df6778944e9c8d1b663004a83903: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:58:22.957087 27805 consensus_queue.cc:1035] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903 [LEADER]: Connected to new peer: Peer: permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:58:22.965423 27580 raft_consensus.cc:1273] T 66b1b69c203145e4bcc2cb26522c663f P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Refusing update from remote peer 15a3df6778944e9c8d1b663004a83903: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:58:22.966629 27738 consensus_queue.cc:1035] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:58:23.823544 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:23.828794 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:23.829137 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:24.829981 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:24.835091 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:24.835443 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:25.836305 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:25.841447 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:25.841809 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:26.842834 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:26.848199 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:26.848606 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:27.849581 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:27.854583 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:27.854957 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:28.855849 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:28.861773 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:28.862167 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:29.863183 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:29.868252 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:29.868623 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:30.869380 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:30.875404 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:30.875890 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:31.904299 26858 auto_leader_rebalancer-test.cc:362] The leader distribution is
I20250905 19:58:31.908403 27654 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "1a6727b25590409b9402c5aee52a55a7"
dest_uuid: "15a3df6778944e9c8d1b663004a83903"
mode: GRACEFUL
new_leader_uuid: "025319773a4247b99829aa83dd002b0f"
from {username='slave'} at 127.0.0.1:55086
I20250905 19:58:31.909049 27654 raft_consensus.cc:604] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903 [term 2 LEADER]: Received request to transfer leadership to TS 025319773a4247b99829aa83dd002b0f
I20250905 19:58:31.915468 27654 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "66b1b69c203145e4bcc2cb26522c663f"
dest_uuid: "15a3df6778944e9c8d1b663004a83903"
mode: GRACEFUL
new_leader_uuid: "025319773a4247b99829aa83dd002b0f"
from {username='slave'} at 127.0.0.1:55086
I20250905 19:58:31.915930 27654 raft_consensus.cc:604] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903 [term 2 LEADER]: Received request to transfer leadership to TS 025319773a4247b99829aa83dd002b0f
I20250905 19:58:31.924034 27580 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7fa1390a6ad647498e7b9459277a60ac"
dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
mode: GRACEFUL
new_leader_uuid: "025319773a4247b99829aa83dd002b0f"
from {username='slave'} at 127.0.0.1:52276
I20250905 19:58:31.924448 27580 raft_consensus.cc:604] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 LEADER]: Received request to transfer leadership to TS 025319773a4247b99829aa83dd002b0f
I20250905 19:58:31.928335 27580 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "ccec5ef5fb0d4383a93e6cf0514aefb8"
dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
mode: GRACEFUL
new_leader_uuid: "025319773a4247b99829aa83dd002b0f"
from {username='slave'} at 127.0.0.1:52276
I20250905 19:58:31.928756 27580 raft_consensus.cc:604] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 LEADER]: Received request to transfer leadership to TS 025319773a4247b99829aa83dd002b0f
I20250905 19:58:31.932474 27580 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "3ad0f154c598458fa48d8fd40ce46171"
dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
mode: GRACEFUL
new_leader_uuid: "025319773a4247b99829aa83dd002b0f"
from {username='slave'} at 127.0.0.1:52276
I20250905 19:58:31.932884 27580 raft_consensus.cc:604] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 LEADER]: Received request to transfer leadership to TS 025319773a4247b99829aa83dd002b0f
I20250905 19:58:31.990487 27815 raft_consensus.cc:991] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8: : Instructing follower 025319773a4247b99829aa83dd002b0f to start an election
I20250905 19:58:31.991376 27817 raft_consensus.cc:1079] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 LEADER]: Signalling peer 025319773a4247b99829aa83dd002b0f to start an election
I20250905 19:58:31.992693 27505 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "3ad0f154c598458fa48d8fd40ce46171"
dest_uuid: "025319773a4247b99829aa83dd002b0f"
from {username='slave'} at 127.0.0.1:40184
I20250905 19:58:31.993100 27505 raft_consensus.cc:491] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:58:31.993398 27505 raft_consensus.cc:3058] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:31.998442 27505 raft_consensus.cc:513] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:31.999994 27505 leader_election.cc:290] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 3 election: Requested vote from peers 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929), 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:32.001096 27654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3ad0f154c598458fa48d8fd40ce46171" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "15a3df6778944e9c8d1b663004a83903"
I20250905 19:58:32.000929 27580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3ad0f154c598458fa48d8fd40ce46171" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
I20250905 19:58:32.001677 27654 raft_consensus.cc:3058] T 3ad0f154c598458fa48d8fd40ce46171 P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:32.001752 27580 raft_consensus.cc:3053] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:58:32.002087 27580 raft_consensus.cc:738] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7bf97ab6f8494cdd8eb04ef0988476a8, State: Running, Role: LEADER
I20250905 19:58:32.002933 27580 consensus_queue.cc:260] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:32.004359 27580 raft_consensus.cc:3058] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:32.009294 27654 raft_consensus.cc:2466] T 3ad0f154c598458fa48d8fd40ce46171 P 15a3df6778944e9c8d1b663004a83903 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 025319773a4247b99829aa83dd002b0f in term 3.
I20250905 19:58:32.010596 27471 leader_election.cc:304] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025319773a4247b99829aa83dd002b0f, 15a3df6778944e9c8d1b663004a83903; no voters:
I20250905 19:58:32.011482 27580 raft_consensus.cc:2466] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 025319773a4247b99829aa83dd002b0f in term 3.
I20250905 19:58:32.011538 27816 raft_consensus.cc:2802] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:58:32.012980 27816 raft_consensus.cc:695] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [term 3 LEADER]: Becoming Leader. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Running, Role: LEADER
I20250905 19:58:32.014153 27816 consensus_queue.cc:237] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:32.021014 27401 catalog_manager.cc:5582] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f reported cstate change: term changed from 2 to 3, leader changed from 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130) to 025319773a4247b99829aa83dd002b0f (127.26.58.129). New cstate: current_term: 3 leader_uuid: "025319773a4247b99829aa83dd002b0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:32.041828 27818 raft_consensus.cc:991] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8: : Instructing follower 025319773a4247b99829aa83dd002b0f to start an election
I20250905 19:58:32.042237 27817 raft_consensus.cc:1079] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 LEADER]: Signalling peer 025319773a4247b99829aa83dd002b0f to start an election
I20250905 19:58:32.043540 27505 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7fa1390a6ad647498e7b9459277a60ac"
dest_uuid: "025319773a4247b99829aa83dd002b0f"
from {username='slave'} at 127.0.0.1:40184
I20250905 19:58:32.044044 27505 raft_consensus.cc:491] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:58:32.044302 27505 raft_consensus.cc:3058] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:32.049059 27505 raft_consensus.cc:513] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:32.050257 27505 leader_election.cc:290] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 2 election: Requested vote from peers 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929), 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:32.050983 27580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fa1390a6ad647498e7b9459277a60ac" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
I20250905 19:58:32.051285 27654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fa1390a6ad647498e7b9459277a60ac" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "15a3df6778944e9c8d1b663004a83903"
I20250905 19:58:32.051534 27580 raft_consensus.cc:3053] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:58:32.051772 27654 raft_consensus.cc:3058] T 7fa1390a6ad647498e7b9459277a60ac P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:32.051844 27580 raft_consensus.cc:738] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7bf97ab6f8494cdd8eb04ef0988476a8, State: Running, Role: LEADER
I20250905 19:58:32.052687 27580 consensus_queue.cc:260] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:32.053802 27580 raft_consensus.cc:3058] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:32.058132 27654 raft_consensus.cc:2466] T 7fa1390a6ad647498e7b9459277a60ac P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 025319773a4247b99829aa83dd002b0f in term 2.
I20250905 19:58:32.059177 27471 leader_election.cc:304] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025319773a4247b99829aa83dd002b0f, 15a3df6778944e9c8d1b663004a83903; no voters:
I20250905 19:58:32.059944 27816 raft_consensus.cc:2802] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:58:32.060070 27580 raft_consensus.cc:2466] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 025319773a4247b99829aa83dd002b0f in term 2.
I20250905 19:58:32.060335 27816 raft_consensus.cc:695] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f [term 2 LEADER]: Becoming Leader. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Running, Role: LEADER
I20250905 19:58:32.061115 27816 consensus_queue.cc:237] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:32.067225 27401 catalog_manager.cc:5582] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f reported cstate change: term changed from 1 to 2, leader changed from 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130) to 025319773a4247b99829aa83dd002b0f (127.26.58.129). New cstate: current_term: 2 leader_uuid: "025319773a4247b99829aa83dd002b0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:32.124514 27805 raft_consensus.cc:991] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903: : Instructing follower 025319773a4247b99829aa83dd002b0f to start an election
I20250905 19:58:32.125026 27826 raft_consensus.cc:1079] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903 [term 2 LEADER]: Signalling peer 025319773a4247b99829aa83dd002b0f to start an election
I20250905 19:58:32.126332 27505 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "1a6727b25590409b9402c5aee52a55a7"
dest_uuid: "025319773a4247b99829aa83dd002b0f"
from {username='slave'} at 127.0.0.1:40178
I20250905 19:58:32.126757 27505 raft_consensus.cc:491] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:58:32.126971 27505 raft_consensus.cc:3058] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:32.131134 27505 raft_consensus.cc:513] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } }
I20250905 19:58:32.132371 27505 leader_election.cc:290] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 3 election: Requested vote from peers 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617), 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929)
I20250905 19:58:32.133245 27654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a6727b25590409b9402c5aee52a55a7" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "15a3df6778944e9c8d1b663004a83903"
I20250905 19:58:32.133325 27580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a6727b25590409b9402c5aee52a55a7" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
I20250905 19:58:32.133684 27654 raft_consensus.cc:3053] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:58:32.133798 27580 raft_consensus.cc:3058] T 1a6727b25590409b9402c5aee52a55a7 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:32.133939 27654 raft_consensus.cc:738] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 15a3df6778944e9c8d1b663004a83903, State: Running, Role: LEADER
I20250905 19:58:32.134582 27654 consensus_queue.cc:260] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } }
I20250905 19:58:32.135455 27654 raft_consensus.cc:3058] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:32.139277 27580 raft_consensus.cc:2466] T 1a6727b25590409b9402c5aee52a55a7 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 025319773a4247b99829aa83dd002b0f in term 3.
I20250905 19:58:32.139420 27654 raft_consensus.cc:2466] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 025319773a4247b99829aa83dd002b0f in term 3.
I20250905 19:58:32.140383 27472 leader_election.cc:304] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025319773a4247b99829aa83dd002b0f, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters:
I20250905 19:58:32.141044 27816 raft_consensus.cc:2802] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:58:32.141373 27816 raft_consensus.cc:695] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [term 3 LEADER]: Becoming Leader. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Running, Role: LEADER
I20250905 19:58:32.142098 27816 consensus_queue.cc:237] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } }
I20250905 19:58:32.148047 27401 catalog_manager.cc:5582] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f reported cstate change: term changed from 2 to 3, leader changed from 15a3df6778944e9c8d1b663004a83903 (127.26.58.131) to 025319773a4247b99829aa83dd002b0f (127.26.58.129). New cstate: current_term: 3 leader_uuid: "025319773a4247b99829aa83dd002b0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } health_report { overall_health: HEALTHY } } }
I20250905 19:58:32.334110 27805 raft_consensus.cc:991] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903: : Instructing follower 025319773a4247b99829aa83dd002b0f to start an election
I20250905 19:58:32.334535 27826 raft_consensus.cc:1079] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903 [term 2 LEADER]: Signalling peer 025319773a4247b99829aa83dd002b0f to start an election
I20250905 19:58:32.335809 27505 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "66b1b69c203145e4bcc2cb26522c663f"
dest_uuid: "025319773a4247b99829aa83dd002b0f"
from {username='slave'} at 127.0.0.1:40178
I20250905 19:58:32.336232 27505 raft_consensus.cc:491] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:58:32.336488 27505 raft_consensus.cc:3058] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:32.340624 27505 raft_consensus.cc:513] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:32.341887 27505 leader_election.cc:290] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 3 election: Requested vote from peers 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929), 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:32.342725 27580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66b1b69c203145e4bcc2cb26522c663f" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
I20250905 19:58:32.342877 27654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66b1b69c203145e4bcc2cb26522c663f" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "15a3df6778944e9c8d1b663004a83903"
I20250905 19:58:32.343163 27580 raft_consensus.cc:3058] T 66b1b69c203145e4bcc2cb26522c663f P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:32.343374 27654 raft_consensus.cc:3053] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:58:32.343641 27654 raft_consensus.cc:738] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 15a3df6778944e9c8d1b663004a83903, State: Running, Role: LEADER
I20250905 19:58:32.344208 27654 consensus_queue.cc:260] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:32.344971 27654 raft_consensus.cc:3058] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:32.347481 27580 raft_consensus.cc:2466] T 66b1b69c203145e4bcc2cb26522c663f P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 025319773a4247b99829aa83dd002b0f in term 3.
I20250905 19:58:32.348603 27472 leader_election.cc:304] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025319773a4247b99829aa83dd002b0f, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters:
I20250905 19:58:32.349331 27816 raft_consensus.cc:2802] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:58:32.349579 27654 raft_consensus.cc:2466] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 025319773a4247b99829aa83dd002b0f in term 3.
I20250905 19:58:32.349702 27816 raft_consensus.cc:695] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [term 3 LEADER]: Becoming Leader. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Running, Role: LEADER
I20250905 19:58:32.350610 27816 consensus_queue.cc:237] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:32.357038 27401 catalog_manager.cc:5582] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f reported cstate change: term changed from 2 to 3, leader changed from 15a3df6778944e9c8d1b663004a83903 (127.26.58.131) to 025319773a4247b99829aa83dd002b0f (127.26.58.129). New cstate: current_term: 3 leader_uuid: "025319773a4247b99829aa83dd002b0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:32.374564 27818 raft_consensus.cc:991] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8: : Instructing follower 025319773a4247b99829aa83dd002b0f to start an election
I20250905 19:58:32.375012 27817 raft_consensus.cc:1079] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 LEADER]: Signalling peer 025319773a4247b99829aa83dd002b0f to start an election
I20250905 19:58:32.376276 27505 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "ccec5ef5fb0d4383a93e6cf0514aefb8"
dest_uuid: "025319773a4247b99829aa83dd002b0f"
from {username='slave'} at 127.0.0.1:40184
I20250905 19:58:32.376653 27505 raft_consensus.cc:491] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:58:32.376883 27505 raft_consensus.cc:3058] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:32.381031 27505 raft_consensus.cc:513] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:32.382295 27505 leader_election.cc:290] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 3 election: Requested vote from peers 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929), 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:32.383098 27580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccec5ef5fb0d4383a93e6cf0514aefb8" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
I20250905 19:58:32.383232 27654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccec5ef5fb0d4383a93e6cf0514aefb8" candidate_uuid: "025319773a4247b99829aa83dd002b0f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "15a3df6778944e9c8d1b663004a83903"
I20250905 19:58:32.383545 27580 raft_consensus.cc:3053] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:58:32.383661 27654 raft_consensus.cc:3058] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:32.383801 27580 raft_consensus.cc:738] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7bf97ab6f8494cdd8eb04ef0988476a8, State: Running, Role: LEADER
I20250905 19:58:32.384464 27580 consensus_queue.cc:260] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:32.385267 27580 raft_consensus.cc:3058] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:32.387670 27654 raft_consensus.cc:2466] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 025319773a4247b99829aa83dd002b0f in term 3.
I20250905 19:58:32.388505 27471 leader_election.cc:304] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025319773a4247b99829aa83dd002b0f, 15a3df6778944e9c8d1b663004a83903; no voters:
I20250905 19:58:32.389155 27816 raft_consensus.cc:2802] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:58:32.389456 27580 raft_consensus.cc:2466] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 025319773a4247b99829aa83dd002b0f in term 3.
I20250905 19:58:32.389518 27816 raft_consensus.cc:695] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [term 3 LEADER]: Becoming Leader. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Running, Role: LEADER
I20250905 19:58:32.390597 27816 consensus_queue.cc:237] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:32.396888 27401 catalog_manager.cc:5582] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f reported cstate change: term changed from 2 to 3, leader changed from 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130) to 025319773a4247b99829aa83dd002b0f (127.26.58.129). New cstate: current_term: 3 leader_uuid: "025319773a4247b99829aa83dd002b0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:32.414955 27654 raft_consensus.cc:1273] T 3ad0f154c598458fa48d8fd40ce46171 P 15a3df6778944e9c8d1b663004a83903 [term 3 FOLLOWER]: Refusing update from remote peer 025319773a4247b99829aa83dd002b0f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:58:32.416008 27816 consensus_queue.cc:1035] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:58:32.425328 27580 raft_consensus.cc:1273] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 3 FOLLOWER]: Refusing update from remote peer 025319773a4247b99829aa83dd002b0f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:58:32.426843 27816 consensus_queue.cc:1035] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:58:32.533025 27580 raft_consensus.cc:1273] T 1a6727b25590409b9402c5aee52a55a7 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 3 FOLLOWER]: Refusing update from remote peer 025319773a4247b99829aa83dd002b0f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:58:32.534170 27816 consensus_queue.cc:1035] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:58:32.542626 27654 raft_consensus.cc:1273] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903 [term 3 FOLLOWER]: Refusing update from remote peer 025319773a4247b99829aa83dd002b0f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:58:32.544937 27819 consensus_queue.cc:1035] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:58:32.610574 27580 raft_consensus.cc:1273] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Refusing update from remote peer 025319773a4247b99829aa83dd002b0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:58:32.611591 27819 consensus_queue.cc:1035] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:58:32.620568 27654 raft_consensus.cc:1273] T 7fa1390a6ad647498e7b9459277a60ac P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Refusing update from remote peer 025319773a4247b99829aa83dd002b0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:58:32.622367 27837 consensus_queue.cc:1035] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:58:32.793804 27654 raft_consensus.cc:1273] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [term 3 FOLLOWER]: Refusing update from remote peer 025319773a4247b99829aa83dd002b0f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:58:32.794865 27837 consensus_queue.cc:1035] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:58:32.803000 27580 raft_consensus.cc:1273] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 3 FOLLOWER]: Refusing update from remote peer 025319773a4247b99829aa83dd002b0f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:58:32.804905 27836 consensus_queue.cc:1035] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:58:32.887996 27580 raft_consensus.cc:1273] T 66b1b69c203145e4bcc2cb26522c663f P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 3 FOLLOWER]: Refusing update from remote peer 025319773a4247b99829aa83dd002b0f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:58:32.888430 27654 raft_consensus.cc:1273] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903 [term 3 FOLLOWER]: Refusing update from remote peer 025319773a4247b99829aa83dd002b0f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:58:32.889098 27836 consensus_queue.cc:1035] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:58:32.889696 27816 consensus_queue.cc:1035] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:58:34.965564 26858 auto_leader_rebalancer-test.cc:377] The leader distribution is
I20250905 19:58:34.966162 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:34.972896 27505 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5eb30fb918af423aba649dba5b74f749"
dest_uuid: "025319773a4247b99829aa83dd002b0f"
mode: GRACEFUL
new_leader_uuid: "15a3df6778944e9c8d1b663004a83903"
from {username='slave'} at 127.0.0.1:40198
I20250905 19:58:34.973412 27505 raft_consensus.cc:604] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [term 2 LEADER]: Received request to transfer leadership to TS 15a3df6778944e9c8d1b663004a83903
I20250905 19:58:34.975637 27505 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7fa1390a6ad647498e7b9459277a60ac"
dest_uuid: "025319773a4247b99829aa83dd002b0f"
mode: GRACEFUL
new_leader_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
from {username='slave'} at 127.0.0.1:40198
I20250905 19:58:34.976150 27505 raft_consensus.cc:604] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f [term 2 LEADER]: Received request to transfer leadership to TS 7bf97ab6f8494cdd8eb04ef0988476a8
I20250905 19:58:34.977982 27505 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "ccec5ef5fb0d4383a93e6cf0514aefb8"
dest_uuid: "025319773a4247b99829aa83dd002b0f"
mode: GRACEFUL
new_leader_uuid: "15a3df6778944e9c8d1b663004a83903"
from {username='slave'} at 127.0.0.1:40198
I20250905 19:58:34.978356 27505 raft_consensus.cc:604] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [term 3 LEADER]: Received request to transfer leadership to TS 15a3df6778944e9c8d1b663004a83903
I20250905 19:58:34.980161 27505 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "fa10aea786b5462c82a92fc0f9bff996"
dest_uuid: "025319773a4247b99829aa83dd002b0f"
mode: GRACEFUL
new_leader_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
from {username='slave'} at 127.0.0.1:40198
I20250905 19:58:34.980561 27505 raft_consensus.cc:604] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f [term 2 LEADER]: Received request to transfer leadership to TS 7bf97ab6f8494cdd8eb04ef0988476a8
I20250905 19:58:34.981472 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250905 19:58:34.981920 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:35.067394 27819 raft_consensus.cc:991] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f: : Instructing follower 7bf97ab6f8494cdd8eb04ef0988476a8 to start an election
I20250905 19:58:35.067837 27845 raft_consensus.cc:1079] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f [term 2 LEADER]: Signalling peer 7bf97ab6f8494cdd8eb04ef0988476a8 to start an election
I20250905 19:58:35.069110 27581 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7fa1390a6ad647498e7b9459277a60ac"
dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
from {username='slave'} at 127.0.0.1:52260
I20250905 19:58:35.069548 27581 raft_consensus.cc:491] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:58:35.069789 27581 raft_consensus.cc:3058] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:35.074193 27581 raft_consensus.cc:513] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:35.075943 27581 leader_election.cc:290] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 3 election: Requested vote from peers 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725), 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:35.076918 27505 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fa1390a6ad647498e7b9459277a60ac" candidate_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "025319773a4247b99829aa83dd002b0f"
I20250905 19:58:35.076996 27654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fa1390a6ad647498e7b9459277a60ac" candidate_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "15a3df6778944e9c8d1b663004a83903"
I20250905 19:58:35.077513 27505 raft_consensus.cc:3053] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:58:35.077553 27654 raft_consensus.cc:3058] T 7fa1390a6ad647498e7b9459277a60ac P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:35.077792 27505 raft_consensus.cc:738] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Running, Role: LEADER
I20250905 19:58:35.078395 27505 consensus_queue.cc:260] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:35.079255 27505 raft_consensus.cc:3058] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:35.081868 27654 raft_consensus.cc:2466] T 7fa1390a6ad647498e7b9459277a60ac P 15a3df6778944e9c8d1b663004a83903 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bf97ab6f8494cdd8eb04ef0988476a8 in term 3.
I20250905 19:58:35.082893 27546 leader_election.cc:304] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15a3df6778944e9c8d1b663004a83903, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters:
I20250905 19:58:35.083524 27817 raft_consensus.cc:2802] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:58:35.083547 27505 raft_consensus.cc:2466] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bf97ab6f8494cdd8eb04ef0988476a8 in term 3.
I20250905 19:58:35.084540 27817 raft_consensus.cc:695] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 3 LEADER]: Becoming Leader. State: Replica: 7bf97ab6f8494cdd8eb04ef0988476a8, State: Running, Role: LEADER
I20250905 19:58:35.085212 27817 consensus_queue.cc:237] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:35.091602 27400 catalog_manager.cc:5582] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 reported cstate change: term changed from 2 to 3, leader changed from 025319773a4247b99829aa83dd002b0f (127.26.58.129) to 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130). New cstate: current_term: 3 leader_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:35.116807 27819 raft_consensus.cc:991] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f: : Instructing follower 7bf97ab6f8494cdd8eb04ef0988476a8 to start an election
I20250905 19:58:35.117170 27845 raft_consensus.cc:1079] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f [term 2 LEADER]: Signalling peer 7bf97ab6f8494cdd8eb04ef0988476a8 to start an election
I20250905 19:58:35.118301 27581 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "fa10aea786b5462c82a92fc0f9bff996"
dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
from {username='slave'} at 127.0.0.1:52260
I20250905 19:58:35.118698 27581 raft_consensus.cc:491] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:58:35.118917 27581 raft_consensus.cc:3058] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:35.123479 27581 raft_consensus.cc:513] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:35.124771 27581 leader_election.cc:290] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 3 election: Requested vote from peers 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725), 15a3df6778944e9c8d1b663004a83903 (127.26.58.131:42617)
I20250905 19:58:35.125453 27505 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa10aea786b5462c82a92fc0f9bff996" candidate_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "025319773a4247b99829aa83dd002b0f"
I20250905 19:58:35.125680 27654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa10aea786b5462c82a92fc0f9bff996" candidate_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "15a3df6778944e9c8d1b663004a83903"
I20250905 19:58:35.125942 27505 raft_consensus.cc:3053] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:58:35.126116 27654 raft_consensus.cc:3058] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:35.126230 27505 raft_consensus.cc:738] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Running, Role: LEADER
I20250905 19:58:35.126919 27505 consensus_queue.cc:260] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:35.127832 27505 raft_consensus.cc:3058] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:35.131137 27654 raft_consensus.cc:2466] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bf97ab6f8494cdd8eb04ef0988476a8 in term 3.
I20250905 19:58:35.131911 27546 leader_election.cc:304] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15a3df6778944e9c8d1b663004a83903, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters:
I20250905 19:58:35.132536 27817 raft_consensus.cc:2802] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:58:35.132572 27505 raft_consensus.cc:2466] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bf97ab6f8494cdd8eb04ef0988476a8 in term 3.
I20250905 19:58:35.132977 27817 raft_consensus.cc:695] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 3 LEADER]: Becoming Leader. State: Replica: 7bf97ab6f8494cdd8eb04ef0988476a8, State: Running, Role: LEADER
I20250905 19:58:35.133847 27817 consensus_queue.cc:237] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:35.140277 27400 catalog_manager.cc:5582] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8 reported cstate change: term changed from 2 to 3, leader changed from 025319773a4247b99829aa83dd002b0f (127.26.58.129) to 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130). New cstate: current_term: 3 leader_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:35.148373 27819 raft_consensus.cc:991] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f: : Instructing follower 15a3df6778944e9c8d1b663004a83903 to start an election
I20250905 19:58:35.148741 27845 raft_consensus.cc:1079] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [term 2 LEADER]: Signalling peer 15a3df6778944e9c8d1b663004a83903 to start an election
I20250905 19:58:35.149868 27654 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "5eb30fb918af423aba649dba5b74f749"
dest_uuid: "15a3df6778944e9c8d1b663004a83903"
from {username='slave'} at 127.0.0.1:55072
I20250905 19:58:35.150265 27654 raft_consensus.cc:491] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:58:35.150557 27654 raft_consensus.cc:3058] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:35.154477 27654 raft_consensus.cc:513] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:35.156044 27654 leader_election.cc:290] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903 [CANDIDATE]: Term 3 election: Requested vote from peers 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725), 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929)
I20250905 19:58:35.156842 27505 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eb30fb918af423aba649dba5b74f749" candidate_uuid: "15a3df6778944e9c8d1b663004a83903" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "025319773a4247b99829aa83dd002b0f"
I20250905 19:58:35.157029 27581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5eb30fb918af423aba649dba5b74f749" candidate_uuid: "15a3df6778944e9c8d1b663004a83903" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
I20250905 19:58:35.157274 27505 raft_consensus.cc:3053] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:58:35.157486 27581 raft_consensus.cc:3058] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:35.157531 27505 raft_consensus.cc:738] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Running, Role: LEADER
I20250905 19:58:35.158221 27505 consensus_queue.cc:260] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:35.159053 27505 raft_consensus.cc:3058] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:35.161592 27581 raft_consensus.cc:2466] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15a3df6778944e9c8d1b663004a83903 in term 3.
I20250905 19:58:35.162600 27621 leader_election.cc:304] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15a3df6778944e9c8d1b663004a83903, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters:
I20250905 19:58:35.163288 27505 raft_consensus.cc:2466] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15a3df6778944e9c8d1b663004a83903 in term 3.
I20250905 19:58:35.163323 27826 raft_consensus.cc:2802] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:58:35.164462 27826 raft_consensus.cc:695] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903 [term 3 LEADER]: Becoming Leader. State: Replica: 15a3df6778944e9c8d1b663004a83903, State: Running, Role: LEADER
I20250905 19:58:35.165122 27826 consensus_queue.cc:237] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:35.171916 27400 catalog_manager.cc:5582] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903 reported cstate change: term changed from 2 to 3, leader changed from 025319773a4247b99829aa83dd002b0f (127.26.58.129) to 15a3df6778944e9c8d1b663004a83903 (127.26.58.131). New cstate: current_term: 3 leader_uuid: "15a3df6778944e9c8d1b663004a83903" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } health_report { overall_health: HEALTHY } } }
I20250905 19:58:35.266846 27819 raft_consensus.cc:991] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f: : Instructing follower 15a3df6778944e9c8d1b663004a83903 to start an election
I20250905 19:58:35.267275 27845 raft_consensus.cc:1079] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [term 3 LEADER]: Signalling peer 15a3df6778944e9c8d1b663004a83903 to start an election
I20250905 19:58:35.268492 27654 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "ccec5ef5fb0d4383a93e6cf0514aefb8"
dest_uuid: "15a3df6778944e9c8d1b663004a83903"
from {username='slave'} at 127.0.0.1:55072
I20250905 19:58:35.268910 27654 raft_consensus.cc:491] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:58:35.269187 27654 raft_consensus.cc:3058] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:58:35.273270 27654 raft_consensus.cc:513] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:35.274642 27654 leader_election.cc:290] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [CANDIDATE]: Term 4 election: Requested vote from peers 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725), 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929)
I20250905 19:58:35.275410 27505 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccec5ef5fb0d4383a93e6cf0514aefb8" candidate_uuid: "15a3df6778944e9c8d1b663004a83903" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "025319773a4247b99829aa83dd002b0f"
I20250905 19:58:35.275604 27581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccec5ef5fb0d4383a93e6cf0514aefb8" candidate_uuid: "15a3df6778944e9c8d1b663004a83903" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8"
I20250905 19:58:35.275895 27505 raft_consensus.cc:3053] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:58:35.276046 27581 raft_consensus.cc:3058] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:58:35.276161 27505 raft_consensus.cc:738] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 025319773a4247b99829aa83dd002b0f, State: Running, Role: LEADER
I20250905 19:58:35.276815 27505 consensus_queue.cc:260] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:35.277652 27505 raft_consensus.cc:3058] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:58:35.280304 27581 raft_consensus.cc:2466] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15a3df6778944e9c8d1b663004a83903 in term 4.
I20250905 19:58:35.281139 27621 leader_election.cc:304] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15a3df6778944e9c8d1b663004a83903, 7bf97ab6f8494cdd8eb04ef0988476a8; no voters:
I20250905 19:58:35.281806 27826 raft_consensus.cc:2802] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:58:35.281910 27505 raft_consensus.cc:2466] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15a3df6778944e9c8d1b663004a83903 in term 4.
I20250905 19:58:35.282225 27826 raft_consensus.cc:695] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [term 4 LEADER]: Becoming Leader. State: Replica: 15a3df6778944e9c8d1b663004a83903, State: Running, Role: LEADER
I20250905 19:58:35.283051 27826 consensus_queue.cc:237] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } }
I20250905 19:58:35.288967 27400 catalog_manager.cc:5582] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 reported cstate change: term changed from 3 to 4, leader changed from 025319773a4247b99829aa83dd002b0f (127.26.58.129) to 15a3df6778944e9c8d1b663004a83903 (127.26.58.131). New cstate: current_term: 4 leader_uuid: "15a3df6778944e9c8d1b663004a83903" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 } health_report { overall_health: HEALTHY } } }
I20250905 19:58:35.581122 27654 raft_consensus.cc:1273] T 7fa1390a6ad647498e7b9459277a60ac P 15a3df6778944e9c8d1b663004a83903 [term 3 FOLLOWER]: Refusing update from remote peer 7bf97ab6f8494cdd8eb04ef0988476a8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:58:35.582162 27817 consensus_queue.cc:1035] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:58:35.590297 27505 raft_consensus.cc:1273] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f [term 3 FOLLOWER]: Refusing update from remote peer 7bf97ab6f8494cdd8eb04ef0988476a8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:58:35.591759 27817 consensus_queue.cc:1035] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:58:35.610340 27505 raft_consensus.cc:1273] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f [term 3 FOLLOWER]: Refusing update from remote peer 7bf97ab6f8494cdd8eb04ef0988476a8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:58:35.611349 27817 consensus_queue.cc:1035] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:58:35.614316 27506 raft_consensus.cc:1273] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [term 3 FOLLOWER]: Refusing update from remote peer 15a3df6778944e9c8d1b663004a83903: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:58:35.615711 27826 consensus_queue.cc:1035] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903 [LEADER]: Connected to new peer: Peer: permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:58:35.619639 27654 raft_consensus.cc:1273] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [term 3 FOLLOWER]: Refusing update from remote peer 7bf97ab6f8494cdd8eb04ef0988476a8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:58:35.621431 27862 consensus_queue.cc:1035] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15a3df6778944e9c8d1b663004a83903" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 42617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:58:35.627403 27581 raft_consensus.cc:1273] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 3 FOLLOWER]: Refusing update from remote peer 15a3df6778944e9c8d1b663004a83903: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:58:35.628885 27853 consensus_queue.cc:1035] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:58:35.693558 27505 raft_consensus.cc:1273] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [term 4 FOLLOWER]: Refusing update from remote peer 15a3df6778944e9c8d1b663004a83903: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250905 19:58:35.694494 27853 consensus_queue.cc:1035] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [LEADER]: Connected to new peer: Peer: permanent_uuid: "025319773a4247b99829aa83dd002b0f" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:58:35.702268 27581 raft_consensus.cc:1273] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 4 FOLLOWER]: Refusing update from remote peer 15a3df6778944e9c8d1b663004a83903: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250905 19:58:35.704068 27853 consensus_queue.cc:1035] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf97ab6f8494cdd8eb04ef0988476a8" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 43929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:58:35.982781 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:35.988468 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:35.988868 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:36.989971 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:36.995661 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:36.996125 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:37.997095 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:38.002941 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:38.003379 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:39.004266 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:39.009653 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:39.010155 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:40.011202 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:40.016597 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:40.017057 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:41.017931 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:41.023473 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:41.023927 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:42.024930 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:42.031518 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:42.031983 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:43.033011 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:43.039655 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:43.040158 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:44.041095 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:58:44.046329 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 19:58:44.046852 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:58:45.077191 26858 auto_leader_rebalancer-test.cc:388] The leader distribution is
I20250905 19:58:45.077752 26858 tablet_server.cc:178] TabletServer@127.26.58.129:0 shutting down...
I20250905 19:58:45.106205 26858 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:58:45.106947 26858 tablet_replica.cc:331] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f: stopping tablet replica
I20250905 19:58:45.107666 26858 raft_consensus.cc:2241] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [term 3 LEADER]: Raft consensus shutting down.
I20250905 19:58:45.108572 26858 raft_consensus.cc:2270] T 1a6727b25590409b9402c5aee52a55a7 P 025319773a4247b99829aa83dd002b0f [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:45.110540 26858 tablet_replica.cc:331] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f: stopping tablet replica
I20250905 19:58:45.111044 26858 raft_consensus.cc:2241] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [term 4 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:45.111449 26858 raft_consensus.cc:2270] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 025319773a4247b99829aa83dd002b0f [term 4 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:45.113202 26858 tablet_replica.cc:331] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f: stopping tablet replica
I20250905 19:58:45.113694 26858 raft_consensus.cc:2241] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:45.114192 26858 raft_consensus.cc:2270] T 650634b3e04147809fdde8cf72063c78 P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Raft consensus is shut down!
W20250905 19:58:45.116462 27547 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 4 similar messages]
I20250905 19:58:45.116719 26858 tablet_replica.cc:331] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f: stopping tablet replica
I20250905 19:58:45.117682 26858 raft_consensus.cc:2241] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f [term 3 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:45.118310 26858 raft_consensus.cc:2270] T 7fa1390a6ad647498e7b9459277a60ac P 025319773a4247b99829aa83dd002b0f [term 3 FOLLOWER]: Raft consensus is shut down!
W20250905 19:58:45.120046 27547 consensus_peers.cc:489] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 -> Peer 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725): Couldn't send request to peer 025319773a4247b99829aa83dd002b0f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:58:45.120910 26858 tablet_replica.cc:331] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 025319773a4247b99829aa83dd002b0f: stopping tablet replica
I20250905 19:58:45.121613 26858 raft_consensus.cc:2241] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:45.122115 26858 raft_consensus.cc:2270] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 025319773a4247b99829aa83dd002b0f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:45.124188 26858 tablet_replica.cc:331] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f: stopping tablet replica
I20250905 19:58:45.124699 26858 raft_consensus.cc:2241] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f [term 3 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:45.125146 26858 raft_consensus.cc:2270] T fa10aea786b5462c82a92fc0f9bff996 P 025319773a4247b99829aa83dd002b0f [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:45.126989 26858 tablet_replica.cc:331] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f: stopping tablet replica
I20250905 19:58:45.127465 26858 raft_consensus.cc:2241] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [term 3 LEADER]: Raft consensus shutting down.
I20250905 19:58:45.128255 26858 raft_consensus.cc:2270] T 3ad0f154c598458fa48d8fd40ce46171 P 025319773a4247b99829aa83dd002b0f [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:45.130013 26858 tablet_replica.cc:331] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f: stopping tablet replica
I20250905 19:58:45.130497 26858 raft_consensus.cc:2241] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:58:45.131229 26858 raft_consensus.cc:2270] T 25d5dc671025465b90a24a0f20dae08f P 025319773a4247b99829aa83dd002b0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:45.133018 26858 tablet_replica.cc:331] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f: stopping tablet replica
I20250905 19:58:45.133445 26858 raft_consensus.cc:2241] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [term 3 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:45.133852 26858 raft_consensus.cc:2270] T 5eb30fb918af423aba649dba5b74f749 P 025319773a4247b99829aa83dd002b0f [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:45.135578 26858 tablet_replica.cc:331] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f: stopping tablet replica
I20250905 19:58:45.136029 26858 raft_consensus.cc:2241] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [term 3 LEADER]: Raft consensus shutting down.
I20250905 19:58:45.136889 26858 raft_consensus.cc:2270] T 66b1b69c203145e4bcc2cb26522c663f P 025319773a4247b99829aa83dd002b0f [term 3 FOLLOWER]: Raft consensus is shut down!
W20250905 19:58:45.163329 27547 consensus_peers.cc:489] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8 -> Peer 025319773a4247b99829aa83dd002b0f (127.26.58.129:43725): Couldn't send request to peer 025319773a4247b99829aa83dd002b0f. Status: Network error: Client connection negotiation failed: client connection to 127.26.58.129:43725: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 19:58:45.167986 26858 tablet_server.cc:195] TabletServer@127.26.58.129:0 shutdown complete.
I20250905 19:58:45.189523 26858 tablet_server.cc:178] TabletServer@127.26.58.130:0 shutting down...
I20250905 19:58:45.216238 26858 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:58:45.216985 26858 tablet_replica.cc:331] T 650634b3e04147809fdde8cf72063c78 P 7bf97ab6f8494cdd8eb04ef0988476a8: stopping tablet replica
I20250905 19:58:45.217576 26858 raft_consensus.cc:2241] T 650634b3e04147809fdde8cf72063c78 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 LEADER]: Raft consensus shutting down.
I20250905 19:58:45.218458 26858 raft_consensus.cc:2270] T 650634b3e04147809fdde8cf72063c78 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:45.220326 26858 tablet_replica.cc:331] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8: stopping tablet replica
I20250905 19:58:45.220887 26858 raft_consensus.cc:2241] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:45.221338 26858 raft_consensus.cc:2270] T 5eb30fb918af423aba649dba5b74f749 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:45.223289 26858 tablet_replica.cc:331] T 66b1b69c203145e4bcc2cb26522c663f P 7bf97ab6f8494cdd8eb04ef0988476a8: stopping tablet replica
I20250905 19:58:45.223795 26858 raft_consensus.cc:2241] T 66b1b69c203145e4bcc2cb26522c663f P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:45.224277 26858 raft_consensus.cc:2270] T 66b1b69c203145e4bcc2cb26522c663f P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:45.226064 26858 tablet_replica.cc:331] T 1a6727b25590409b9402c5aee52a55a7 P 7bf97ab6f8494cdd8eb04ef0988476a8: stopping tablet replica
I20250905 19:58:45.226629 26858 raft_consensus.cc:2241] T 1a6727b25590409b9402c5aee52a55a7 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:45.227059 26858 raft_consensus.cc:2270] T 1a6727b25590409b9402c5aee52a55a7 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:45.228829 26858 tablet_replica.cc:331] T 25d5dc671025465b90a24a0f20dae08f P 7bf97ab6f8494cdd8eb04ef0988476a8: stopping tablet replica
I20250905 19:58:45.229339 26858 raft_consensus.cc:2241] T 25d5dc671025465b90a24a0f20dae08f P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:45.229761 26858 raft_consensus.cc:2270] T 25d5dc671025465b90a24a0f20dae08f P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:45.231642 26858 tablet_replica.cc:331] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8: stopping tablet replica
I20250905 19:58:45.232132 26858 raft_consensus.cc:2241] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:45.232563 26858 raft_consensus.cc:2270] T 3ad0f154c598458fa48d8fd40ce46171 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:45.234364 26858 tablet_replica.cc:331] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8: stopping tablet replica
I20250905 19:58:45.234956 26858 raft_consensus.cc:2241] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:45.235420 26858 raft_consensus.cc:2270] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:45.237361 26858 tablet_replica.cc:331] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8: stopping tablet replica
I20250905 19:58:45.237844 26858 raft_consensus.cc:2241] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 3 LEADER]: Raft consensus shutting down.
I20250905 19:58:45.239032 26858 raft_consensus.cc:2270] T 7fa1390a6ad647498e7b9459277a60ac P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:45.241227 26858 tablet_replica.cc:331] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8: stopping tablet replica
I20250905 19:58:45.241745 26858 raft_consensus.cc:2241] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:45.242199 26858 raft_consensus.cc:2270] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:45.244058 26858 tablet_replica.cc:331] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8: stopping tablet replica
I20250905 19:58:45.244530 26858 raft_consensus.cc:2241] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 3 LEADER]: Raft consensus shutting down.
I20250905 19:58:45.245311 26858 raft_consensus.cc:2270] T fa10aea786b5462c82a92fc0f9bff996 P 7bf97ab6f8494cdd8eb04ef0988476a8 [term 3 FOLLOWER]: Raft consensus is shut down!
W20250905 19:58:45.254794 27621 consensus_peers.cc:489] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903 -> Peer 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929): Couldn't send request to peer 7bf97ab6f8494cdd8eb04ef0988476a8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:58:45.280764 26858 tablet_server.cc:195] TabletServer@127.26.58.130:0 shutdown complete.
W20250905 19:58:45.283350 27621 consensus_peers.cc:489] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 -> Peer 7bf97ab6f8494cdd8eb04ef0988476a8 (127.26.58.130:43929): Couldn't send request to peer 7bf97ab6f8494cdd8eb04ef0988476a8. Status: Network error: Client connection negotiation failed: client connection to 127.26.58.130:43929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 19:58:45.302356 26858 tablet_server.cc:178] TabletServer@127.26.58.131:0 shutting down...
I20250905 19:58:45.331832 26858 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 19:58:45.332525 26858 tablet_replica.cc:331] T 650634b3e04147809fdde8cf72063c78 P 15a3df6778944e9c8d1b663004a83903: stopping tablet replica
I20250905 19:58:45.333109 26858 raft_consensus.cc:2241] T 650634b3e04147809fdde8cf72063c78 P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:45.333549 26858 raft_consensus.cc:2270] T 650634b3e04147809fdde8cf72063c78 P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:45.335798 26858 tablet_replica.cc:331] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903: stopping tablet replica
I20250905 19:58:45.336398 26858 raft_consensus.cc:2241] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:45.336894 26858 raft_consensus.cc:2270] T 1a6727b25590409b9402c5aee52a55a7 P 15a3df6778944e9c8d1b663004a83903 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:45.339015 26858 tablet_replica.cc:331] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 15a3df6778944e9c8d1b663004a83903: stopping tablet replica
I20250905 19:58:45.339552 26858 raft_consensus.cc:2241] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 15a3df6778944e9c8d1b663004a83903 [term 2 LEADER]: Raft consensus shutting down.
I20250905 19:58:45.340394 26858 raft_consensus.cc:2270] T 02a7a3e0f7a74cc2a1d2762c05eca777 P 15a3df6778944e9c8d1b663004a83903 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:45.342267 26858 tablet_replica.cc:331] T 7fa1390a6ad647498e7b9459277a60ac P 15a3df6778944e9c8d1b663004a83903: stopping tablet replica
I20250905 19:58:45.342841 26858 raft_consensus.cc:2241] T 7fa1390a6ad647498e7b9459277a60ac P 15a3df6778944e9c8d1b663004a83903 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:45.343286 26858 raft_consensus.cc:2270] T 7fa1390a6ad647498e7b9459277a60ac P 15a3df6778944e9c8d1b663004a83903 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:45.345064 26858 tablet_replica.cc:331] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903: stopping tablet replica
I20250905 19:58:45.345549 26858 raft_consensus.cc:2241] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:45.346012 26858 raft_consensus.cc:2270] T fa10aea786b5462c82a92fc0f9bff996 P 15a3df6778944e9c8d1b663004a83903 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:45.347930 26858 tablet_replica.cc:331] T 3ad0f154c598458fa48d8fd40ce46171 P 15a3df6778944e9c8d1b663004a83903: stopping tablet replica
I20250905 19:58:45.348502 26858 raft_consensus.cc:2241] T 3ad0f154c598458fa48d8fd40ce46171 P 15a3df6778944e9c8d1b663004a83903 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:45.348946 26858 raft_consensus.cc:2270] T 3ad0f154c598458fa48d8fd40ce46171 P 15a3df6778944e9c8d1b663004a83903 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:45.350804 26858 tablet_replica.cc:331] T 25d5dc671025465b90a24a0f20dae08f P 15a3df6778944e9c8d1b663004a83903: stopping tablet replica
I20250905 19:58:45.351295 26858 raft_consensus.cc:2241] T 25d5dc671025465b90a24a0f20dae08f P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:45.351719 26858 raft_consensus.cc:2270] T 25d5dc671025465b90a24a0f20dae08f P 15a3df6778944e9c8d1b663004a83903 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:45.353514 26858 tablet_replica.cc:331] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903: stopping tablet replica
I20250905 19:58:45.354115 26858 raft_consensus.cc:2241] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903 [term 3 LEADER]: Raft consensus shutting down.
I20250905 19:58:45.355101 26858 raft_consensus.cc:2270] T 5eb30fb918af423aba649dba5b74f749 P 15a3df6778944e9c8d1b663004a83903 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:45.356969 26858 tablet_replica.cc:331] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903: stopping tablet replica
I20250905 19:58:45.357424 26858 raft_consensus.cc:2241] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250905 19:58:45.357847 26858 raft_consensus.cc:2270] T 66b1b69c203145e4bcc2cb26522c663f P 15a3df6778944e9c8d1b663004a83903 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:45.359652 26858 tablet_replica.cc:331] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903: stopping tablet replica
I20250905 19:58:45.360078 26858 raft_consensus.cc:2241] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [term 4 LEADER]: Raft consensus shutting down.
I20250905 19:58:45.360845 26858 raft_consensus.cc:2270] T ccec5ef5fb0d4383a93e6cf0514aefb8 P 15a3df6778944e9c8d1b663004a83903 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:45.390674 26858 tablet_server.cc:195] TabletServer@127.26.58.131:0 shutdown complete.
I20250905 19:58:45.411290 26858 master.cc:561] Master@127.26.58.190:43327 shutting down...
I20250905 19:58:45.433866 26858 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 14cfe5353e384054adfb13763c0386d8 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:58:45.434521 26858 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 14cfe5353e384054adfb13763c0386d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:58:45.434836 26858 tablet_replica.cc:331] T 00000000000000000000000000000000 P 14cfe5353e384054adfb13763c0386d8: stopping tablet replica
I20250905 19:58:45.454654 26858 master.cc:583] Master@127.26.58.190:43327 shutdown complete.
025319773a4247b99829aa83dd002b0f 5
15a3df6778944e9c8d1b663004a83903 1
7bf97ab6f8494cdd8eb04ef0988476a8 4
025319773a4247b99829aa83dd002b0f 3
15a3df6778944e9c8d1b663004a83903 3
7bf97ab6f8494cdd8eb04ef0988476a8 4
025319773a4247b99829aa83dd002b0f 8
15a3df6778944e9c8d1b663004a83903 1
7bf97ab6f8494cdd8eb04ef0988476a8 1
025319773a4247b99829aa83dd002b0f 4
15a3df6778944e9c8d1b663004a83903 3
7bf97ab6f8494cdd8eb04ef0988476a8 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28401 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20250905 19:58:45.496006 26858 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.26.58.190:38969
I20250905 19:58:45.497167 26858 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:58:45.503892 27883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:58:45.504086 27884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:58:45.504282 27886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:58:45.505434 26858 server_base.cc:1047] running on GCE node
I20250905 19:58:45.506361 26858 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:58:45.506664 26858 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:58:45.506865 26858 hybrid_clock.cc:648] HybridClock initialized: now 1757102325506846 us; error 0 us; skew 500 ppm
I20250905 19:58:45.507393 26858 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:58:45.509899 26858 webserver.cc:480] Webserver started at http://127.26.58.190:45787/ using document root <none> and password file <none>
I20250905 19:58:45.510413 26858 fs_manager.cc:362] Metadata directory not provided
I20250905 19:58:45.510666 26858 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:58:45.510941 26858 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:58:45.512127 26858 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/master-0-root/instance:
uuid: "44c052c5498545aa93c222520045d410"
format_stamp: "Formatted at 2025-09-05 19:58:45 on dist-test-slave-v26w"
I20250905 19:58:45.516916 26858 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20250905 19:58:45.520583 27891 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:58:45.521476 26858 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 19:58:45.521786 26858 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/master-0-root
uuid: "44c052c5498545aa93c222520045d410"
format_stamp: "Formatted at 2025-09-05 19:58:45 on dist-test-slave-v26w"
I20250905 19:58:45.522065 26858 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:58:45.536649 26858 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:58:45.537922 26858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:58:45.579769 26858 rpc_server.cc:307] RPC server started. Bound to: 127.26.58.190:38969
I20250905 19:58:45.579872 27942 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.58.190:38969 every 8 connection(s)
I20250905 19:58:45.583755 27943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:45.594076 27943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410: Bootstrap starting.
I20250905 19:58:45.599012 27943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:45.603938 27943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410: No bootstrap required, opened a new log
I20250905 19:58:45.606146 27943 raft_consensus.cc:357] T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c052c5498545aa93c222520045d410" member_type: VOTER }
I20250905 19:58:45.606631 27943 raft_consensus.cc:383] T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:45.606897 27943 raft_consensus.cc:738] T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c052c5498545aa93c222520045d410, State: Initialized, Role: FOLLOWER
I20250905 19:58:45.607439 27943 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c052c5498545aa93c222520045d410" member_type: VOTER }
I20250905 19:58:45.607928 27943 raft_consensus.cc:397] T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 19:58:45.608155 27943 raft_consensus.cc:491] T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 19:58:45.608438 27943 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:45.613042 27943 raft_consensus.cc:513] T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c052c5498545aa93c222520045d410" member_type: VOTER }
I20250905 19:58:45.613636 27943 leader_election.cc:304] T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44c052c5498545aa93c222520045d410; no voters:
I20250905 19:58:45.614949 27943 leader_election.cc:290] T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 19:58:45.615298 27946 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:45.616685 27946 raft_consensus.cc:695] T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410 [term 1 LEADER]: Becoming Leader. State: Replica: 44c052c5498545aa93c222520045d410, State: Running, Role: LEADER
I20250905 19:58:45.617451 27946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c052c5498545aa93c222520045d410" member_type: VOTER }
I20250905 19:58:45.618103 27943 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 19:58:45.623277 27948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44c052c5498545aa93c222520045d410. Latest consensus state: current_term: 1 leader_uuid: "44c052c5498545aa93c222520045d410" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c052c5498545aa93c222520045d410" member_type: VOTER } }
I20250905 19:58:45.624182 27948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410 [sys.catalog]: This master's current role is: LEADER
I20250905 19:58:45.623387 27947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44c052c5498545aa93c222520045d410" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c052c5498545aa93c222520045d410" member_type: VOTER } }
I20250905 19:58:45.625494 27947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410 [sys.catalog]: This master's current role is: LEADER
I20250905 19:58:45.631325 27954 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 19:58:45.636636 27954 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 19:58:45.637712 26858 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250905 19:58:45.646718 27954 catalog_manager.cc:1349] Generated new cluster ID: 0e03389acfe849939775fdfa86c87757
I20250905 19:58:45.647038 27954 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 19:58:45.672092 27954 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 19:58:45.673427 27954 catalog_manager.cc:1506] Loading token signing keys...
I20250905 19:58:45.696017 27954 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410: Generated new TSK 0
I20250905 19:58:45.696704 27954 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 19:58:45.704496 26858 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:58:45.710867 27964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:58:45.711992 27965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:58:45.713475 27967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:58:45.714002 26858 server_base.cc:1047] running on GCE node
I20250905 19:58:45.714886 26858 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:58:45.715082 26858 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:58:45.715204 26858 hybrid_clock.cc:648] HybridClock initialized: now 1757102325715193 us; error 0 us; skew 500 ppm
I20250905 19:58:45.715749 26858 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:58:45.718184 26858 webserver.cc:480] Webserver started at http://127.26.58.129:42249/ using document root <none> and password file <none>
I20250905 19:58:45.718755 26858 fs_manager.cc:362] Metadata directory not provided
I20250905 19:58:45.718928 26858 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:58:45.719236 26858 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:58:45.720424 26858 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-0-root/instance:
uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
format_stamp: "Formatted at 2025-09-05 19:58:45 on dist-test-slave-v26w"
I20250905 19:58:45.725675 26858 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250905 19:58:45.729400 27972 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:58:45.730336 26858 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 19:58:45.730731 26858 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-0-root
uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
format_stamp: "Formatted at 2025-09-05 19:58:45 on dist-test-slave-v26w"
I20250905 19:58:45.731021 26858 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:58:45.757007 26858 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:58:45.758324 26858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:58:45.759974 26858 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:58:45.762449 26858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:58:45.762676 26858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:58:45.762945 26858 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:58:45.763130 26858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:58:45.807341 26858 rpc_server.cc:307] RPC server started. Bound to: 127.26.58.129:38421
I20250905 19:58:45.807438 28034 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.58.129:38421 every 8 connection(s)
I20250905 19:58:45.813699 26858 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:58:45.821619 28039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:58:45.822947 28040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:58:45.826884 28042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:58:45.827157 26858 server_base.cc:1047] running on GCE node
I20250905 19:58:45.827649 28035 heartbeater.cc:344] Connected to a master server at 127.26.58.190:38969
I20250905 19:58:45.828027 28035 heartbeater.cc:461] Registering TS with master...
I20250905 19:58:45.828739 26858 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250905 19:58:45.828801 28035 heartbeater.cc:507] Master 127.26.58.190:38969 requested a full tablet report, sending...
W20250905 19:58:45.828996 26858 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:58:45.829216 26858 hybrid_clock.cc:648] HybridClock initialized: now 1757102325829198 us; error 0 us; skew 500 ppm
I20250905 19:58:45.829831 26858 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:58:45.831144 27908 ts_manager.cc:194] Registered new tserver with Master: cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421)
I20250905 19:58:45.832911 26858 webserver.cc:480] Webserver started at http://127.26.58.130:43115/ using document root <none> and password file <none>
I20250905 19:58:45.833283 27908 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34440
I20250905 19:58:45.833446 26858 fs_manager.cc:362] Metadata directory not provided
I20250905 19:58:45.833736 26858 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:58:45.834021 26858 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:58:45.835282 26858 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-1-root/instance:
uuid: "e4413ae5e0b64b52a6f360ac580239b9"
format_stamp: "Formatted at 2025-09-05 19:58:45 on dist-test-slave-v26w"
I20250905 19:58:45.839978 26858 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250905 19:58:45.843750 28047 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:58:45.844707 26858 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250905 19:58:45.845024 26858 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-1-root
uuid: "e4413ae5e0b64b52a6f360ac580239b9"
format_stamp: "Formatted at 2025-09-05 19:58:45 on dist-test-slave-v26w"
I20250905 19:58:45.845325 26858 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:58:45.859745 26858 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:58:45.861011 26858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:58:45.862526 26858 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:58:45.864806 26858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:58:45.865029 26858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:58:45.865260 26858 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:58:45.865414 26858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:58:45.906364 26858 rpc_server.cc:307] RPC server started. Bound to: 127.26.58.130:34193
I20250905 19:58:45.906515 28109 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.58.130:34193 every 8 connection(s)
I20250905 19:58:45.911340 26858 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:58:45.918980 28113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:58:45.920246 28114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:58:45.925242 28116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:58:45.925809 26858 server_base.cc:1047] running on GCE node
I20250905 19:58:45.925913 28110 heartbeater.cc:344] Connected to a master server at 127.26.58.190:38969
I20250905 19:58:45.926396 28110 heartbeater.cc:461] Registering TS with master...
I20250905 19:58:45.926842 26858 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:58:45.927069 26858 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:58:45.927240 26858 hybrid_clock.cc:648] HybridClock initialized: now 1757102325927223 us; error 0 us; skew 500 ppm
I20250905 19:58:45.927196 28110 heartbeater.cc:507] Master 127.26.58.190:38969 requested a full tablet report, sending...
I20250905 19:58:45.927932 26858 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:58:45.929561 27908 ts_manager.cc:194] Registered new tserver with Master: e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:45.930805 26858 webserver.cc:480] Webserver started at http://127.26.58.131:44035/ using document root <none> and password file <none>
I20250905 19:58:45.931192 27908 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34446
I20250905 19:58:45.931345 26858 fs_manager.cc:362] Metadata directory not provided
I20250905 19:58:45.931612 26858 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:58:45.931881 26858 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:58:45.933447 26858 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-2-root/instance:
uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
format_stamp: "Formatted at 2025-09-05 19:58:45 on dist-test-slave-v26w"
I20250905 19:58:45.938238 26858 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20250905 19:58:45.942034 28121 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:58:45.942862 26858 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 19:58:45.943166 26858 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-2-root
uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
format_stamp: "Formatted at 2025-09-05 19:58:45 on dist-test-slave-v26w"
I20250905 19:58:45.943442 26858 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:58:45.961131 26858 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:58:45.962491 26858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:58:45.964152 26858 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:58:45.966694 26858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:58:45.966912 26858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:58:45.967103 26858 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:58:45.967231 26858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:58:46.010550 26858 rpc_server.cc:307] RPC server started. Bound to: 127.26.58.131:36705
I20250905 19:58:46.010694 28183 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.58.131:36705 every 8 connection(s)
I20250905 19:58:46.025223 28184 heartbeater.cc:344] Connected to a master server at 127.26.58.190:38969
I20250905 19:58:46.025648 28184 heartbeater.cc:461] Registering TS with master...
I20250905 19:58:46.026515 28184 heartbeater.cc:507] Master 127.26.58.190:38969 requested a full tablet report, sending...
I20250905 19:58:46.029101 27908 ts_manager.cc:194] Registered new tserver with Master: 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:46.029263 26858 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015575982s
I20250905 19:58:46.031134 27908 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34448
I20250905 19:58:46.032218 26858 test_util.cc:276] Using random seed: 1140467518
I20250905 19:58:46.058576 27908 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34464:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:58:46.064844 27908 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:58:46.334619 28000 tablet_service.cc:1468] Processing CreateTablet for tablet bbd67b7a3c3244838ec2d727f746cc0b (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250905 19:58:46.335532 27996 tablet_service.cc:1468] Processing CreateTablet for tablet 8afd77a8d80843aa805f971a2806ab86 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250905 19:58:46.337880 28000 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbd67b7a3c3244838ec2d727f746cc0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.338373 27997 tablet_service.cc:1468] Processing CreateTablet for tablet 36e45650becf4b5395e131ca159c32c4 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250905 19:58:46.338752 28149 tablet_service.cc:1468] Processing CreateTablet for tablet bbd67b7a3c3244838ec2d727f746cc0b (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250905 19:58:46.340086 27997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e45650becf4b5395e131ca159c32c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.340020 28146 tablet_service.cc:1468] Processing CreateTablet for tablet 36e45650becf4b5395e131ca159c32c4 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250905 19:58:46.341774 28143 tablet_service.cc:1468] Processing CreateTablet for tablet 0c1867b70e8143a5b3d46d269c9c139c (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250905 19:58:46.342363 28142 tablet_service.cc:1468] Processing CreateTablet for tablet 6dbd54765afc49cb862b267d6663000c (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250905 19:58:46.342998 28144 tablet_service.cc:1468] Processing CreateTablet for tablet 2e578d82f2b643a795b79abae746575e (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250905 19:58:46.343860 27998 tablet_service.cc:1468] Processing CreateTablet for tablet 986c4f475a8d46f39465775db70416d3 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250905 19:58:46.344741 28148 tablet_service.cc:1468] Processing CreateTablet for tablet 9f7f6e4e3a754dcda90928621168fbe7 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250905 19:58:46.345448 27998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 986c4f475a8d46f39465775db70416d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.346197 28141 tablet_service.cc:1468] Processing CreateTablet for tablet f2de38aacf1b4dccaeee999400fb6d75 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250905 19:58:46.347812 28145 tablet_service.cc:1468] Processing CreateTablet for tablet 8afd77a8d80843aa805f971a2806ab86 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250905 19:58:46.347867 27999 tablet_service.cc:1468] Processing CreateTablet for tablet 9f7f6e4e3a754dcda90928621168fbe7 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250905 19:58:46.350111 28147 tablet_service.cc:1468] Processing CreateTablet for tablet 986c4f475a8d46f39465775db70416d3 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250905 19:58:46.350131 28140 tablet_service.cc:1468] Processing CreateTablet for tablet e707b0f561934674947fb463ce626a07 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250905 19:58:46.351745 27993 tablet_service.cc:1468] Processing CreateTablet for tablet 6dbd54765afc49cb862b267d6663000c (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250905 19:58:46.352257 27995 tablet_service.cc:1468] Processing CreateTablet for tablet 2e578d82f2b643a795b79abae746575e (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250905 19:58:46.353180 28149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbd67b7a3c3244838ec2d727f746cc0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.355028 27992 tablet_service.cc:1468] Processing CreateTablet for tablet f2de38aacf1b4dccaeee999400fb6d75 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250905 19:58:46.351753 27994 tablet_service.cc:1468] Processing CreateTablet for tablet 0c1867b70e8143a5b3d46d269c9c139c (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250905 19:58:46.355252 27991 tablet_service.cc:1468] Processing CreateTablet for tablet e707b0f561934674947fb463ce626a07 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250905 19:58:46.357573 28146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e45650becf4b5395e131ca159c32c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.359791 28141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2de38aacf1b4dccaeee999400fb6d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.362928 28148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f7f6e4e3a754dcda90928621168fbe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.363741 28143 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c1867b70e8143a5b3d46d269c9c139c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.364691 28140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e707b0f561934674947fb463ce626a07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.365245 27994 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c1867b70e8143a5b3d46d269c9c139c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.367890 27991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e707b0f561934674947fb463ce626a07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.370846 28145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8afd77a8d80843aa805f971a2806ab86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.371032 27992 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2de38aacf1b4dccaeee999400fb6d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.374981 27995 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e578d82f2b643a795b79abae746575e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.377048 28147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 986c4f475a8d46f39465775db70416d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.379890 28075 tablet_service.cc:1468] Processing CreateTablet for tablet bbd67b7a3c3244838ec2d727f746cc0b (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250905 19:58:46.381688 28075 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbd67b7a3c3244838ec2d727f746cc0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.382927 27999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f7f6e4e3a754dcda90928621168fbe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.383893 28074 tablet_service.cc:1468] Processing CreateTablet for tablet 9f7f6e4e3a754dcda90928621168fbe7 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250905 19:58:46.385412 28074 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f7f6e4e3a754dcda90928621168fbe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.386930 27996 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8afd77a8d80843aa805f971a2806ab86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.387962 28073 tablet_service.cc:1468] Processing CreateTablet for tablet 986c4f475a8d46f39465775db70416d3 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250905 19:58:46.391983 27993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dbd54765afc49cb862b267d6663000c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.391821 28071 tablet_service.cc:1468] Processing CreateTablet for tablet 8afd77a8d80843aa805f971a2806ab86 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250905 19:58:46.394419 28070 tablet_service.cc:1468] Processing CreateTablet for tablet 2e578d82f2b643a795b79abae746575e (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250905 19:58:46.397569 28069 tablet_service.cc:1468] Processing CreateTablet for tablet 0c1867b70e8143a5b3d46d269c9c139c (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250905 19:58:46.401014 28068 tablet_service.cc:1468] Processing CreateTablet for tablet 6dbd54765afc49cb862b267d6663000c (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250905 19:58:46.402679 28072 tablet_service.cc:1468] Processing CreateTablet for tablet 36e45650becf4b5395e131ca159c32c4 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250905 19:58:46.406056 28067 tablet_service.cc:1468] Processing CreateTablet for tablet f2de38aacf1b4dccaeee999400fb6d75 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250905 19:58:46.409334 28066 tablet_service.cc:1468] Processing CreateTablet for tablet e707b0f561934674947fb463ce626a07 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250905 19:58:46.412716 28200 tablet_bootstrap.cc:492] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:46.415940 28149 tablet_service.cc:1468] Processing CreateTablet for tablet 0adebfdecbae47298f234b849fcea7fc (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250905 19:58:46.417394 28149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0adebfdecbae47298f234b849fcea7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.419263 28142 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dbd54765afc49cb862b267d6663000c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.420563 27998 tablet_service.cc:1468] Processing CreateTablet for tablet 6c423052d6454829bccfe7c8bc1bda5c (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250905 19:58:46.421973 27998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c423052d6454829bccfe7c8bc1bda5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.413763 28146 tablet_service.cc:1468] Processing CreateTablet for tablet 6c423052d6454829bccfe7c8bc1bda5c (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250905 19:58:46.423461 28144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e578d82f2b643a795b79abae746575e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.425274 28146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c423052d6454829bccfe7c8bc1bda5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.437405 28200 tablet_bootstrap.cc:654] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:46.443259 28070 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e578d82f2b643a795b79abae746575e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.477195 27998 tablet_service.cc:1468] Processing CreateTablet for tablet fc7dc431504743d79dbdc8eea83f9dcb (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250905 19:58:46.478749 27998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc7dc431504743d79dbdc8eea83f9dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.487764 28146 tablet_service.cc:1468] Processing CreateTablet for tablet fc7dc431504743d79dbdc8eea83f9dcb (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250905 19:58:46.489305 28146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc7dc431504743d79dbdc8eea83f9dcb. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:58:46.498740 27895 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 68317.4us
I20250905 19:58:46.515094 28066 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e707b0f561934674947fb463ce626a07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.516384 27998 tablet_service.cc:1468] Processing CreateTablet for tablet 761b89d47d5543d89dd47af039cfe7de (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250905 19:58:46.517794 27998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 761b89d47d5543d89dd47af039cfe7de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.519038 28073 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 986c4f475a8d46f39465775db70416d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.521101 28146 tablet_service.cc:1468] Processing CreateTablet for tablet 761b89d47d5543d89dd47af039cfe7de (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250905 19:58:46.522776 28146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 761b89d47d5543d89dd47af039cfe7de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.523895 28071 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8afd77a8d80843aa805f971a2806ab86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.526096 28069 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c1867b70e8143a5b3d46d269c9c139c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.528326 28072 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e45650becf4b5395e131ca159c32c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.530977 28067 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2de38aacf1b4dccaeee999400fb6d75. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:58:46.574777 27894 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 86296us
I20250905 19:58:46.586680 28068 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dbd54765afc49cb862b267d6663000c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.609714 27999 tablet_service.cc:1468] Processing CreateTablet for tablet 0adebfdecbae47298f234b849fcea7fc (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250905 19:58:46.611222 27999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0adebfdecbae47298f234b849fcea7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.606168 28148 tablet_service.cc:1468] Processing CreateTablet for tablet 62ccdb07048e4762b1b31c36b14610f5 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250905 19:58:46.615554 28148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62ccdb07048e4762b1b31c36b14610f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.616552 28147 tablet_service.cc:1468] Processing CreateTablet for tablet fc6a8d5d253643cea8582bef1c4173d6 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20250905 19:58:46.617113 28072 tablet_service.cc:1468] Processing CreateTablet for tablet 6c423052d6454829bccfe7c8bc1bda5c (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250905 19:58:46.618013 28147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc6a8d5d253643cea8582bef1c4173d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.622669 27996 tablet_service.cc:1468] Processing CreateTablet for tablet b8a1b71a731e455da7990629d2c94801 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250905 19:58:46.624387 27996 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8a1b71a731e455da7990629d2c94801. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:58:46.631927 27956 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250905 19:58:46.631614 28140 tablet_service.cc:1468] Processing CreateTablet for tablet b8a1b71a731e455da7990629d2c94801 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250905 19:58:46.633471 28140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8a1b71a731e455da7990629d2c94801. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.665822 28201 tablet_bootstrap.cc:492] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:46.620095 28072 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c423052d6454829bccfe7c8bc1bda5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.676921 28202 tablet_bootstrap.cc:492] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:46.680972 27997 tablet_service.cc:1468] Processing CreateTablet for tablet fc6a8d5d253643cea8582bef1c4173d6 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20250905 19:58:46.684389 28200 tablet_bootstrap.cc:492] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:46.685037 28200 ts_tablet_manager.cc:1397] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.273s user 0.014s sys 0.024s
I20250905 19:58:46.686559 27997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc6a8d5d253643cea8582bef1c4173d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.687427 28149 tablet_service.cc:1468] Processing CreateTablet for tablet b13af7609cd04cf5b21745527b70074d (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250905 19:58:46.688848 28149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b13af7609cd04cf5b21745527b70074d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.693305 28201 tablet_bootstrap.cc:654] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:46.726032 28000 tablet_service.cc:1468] Processing CreateTablet for tablet 361ffd5c9a994e0397e6748827466d3d (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250905 19:58:46.728331 28000 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 361ffd5c9a994e0397e6748827466d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.728242 28200 raft_consensus.cc:357] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 19:58:46.722666 28067 tablet_service.cc:1468] Processing CreateTablet for tablet b8a1b71a731e455da7990629d2c94801 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250905 19:58:46.705751 27993 tablet_service.cc:1468] Processing CreateTablet for tablet b13af7609cd04cf5b21745527b70074d (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250905 19:58:46.740620 28070 tablet_service.cc:1468] Processing CreateTablet for tablet 62ccdb07048e4762b1b31c36b14610f5 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250905 19:58:46.742000 28067 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8a1b71a731e455da7990629d2c94801. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.721856 28144 tablet_service.cc:1468] Processing CreateTablet for tablet 456d9fe0918b423abeab5c39372bcbe9 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250905 19:58:46.743294 28144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 456d9fe0918b423abeab5c39372bcbe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.747654 27999 tablet_service.cc:1468] Processing CreateTablet for tablet 2f356a2522e646889c023d4b3d26f9cc (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250905 19:58:46.748278 28142 tablet_service.cc:1468] Processing CreateTablet for tablet 2bd6bb972f14472a89ccf99de880efc2 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250905 19:58:46.749817 28142 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bd6bb972f14472a89ccf99de880efc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.707089 28143 tablet_service.cc:1468] Processing CreateTablet for tablet 361ffd5c9a994e0397e6748827466d3d (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250905 19:58:46.755019 28075 tablet_service.cc:1468] Processing CreateTablet for tablet 761b89d47d5543d89dd47af039cfe7de (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250905 19:58:46.756461 28075 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 761b89d47d5543d89dd47af039cfe7de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.756599 28201 tablet_bootstrap.cc:492] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:46.757244 28201 ts_tablet_manager.cc:1397] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.092s user 0.013s sys 0.000s
I20250905 19:58:46.708920 28202 tablet_bootstrap.cc:654] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:46.710572 28074 tablet_service.cc:1468] Processing CreateTablet for tablet 0adebfdecbae47298f234b849fcea7fc (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250905 19:58:46.758689 28074 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0adebfdecbae47298f234b849fcea7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.759346 28145 tablet_service.cc:1468] Processing CreateTablet for tablet 2f356a2522e646889c023d4b3d26f9cc (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250905 19:58:46.699541 28069 tablet_service.cc:1468] Processing CreateTablet for tablet fc7dc431504743d79dbdc8eea83f9dcb (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250905 19:58:46.707044 27992 tablet_service.cc:1468] Processing CreateTablet for tablet 62ccdb07048e4762b1b31c36b14610f5 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250905 19:58:46.763468 28069 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc7dc431504743d79dbdc8eea83f9dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.758737 27993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b13af7609cd04cf5b21745527b70074d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.729497 28200 raft_consensus.cc:383] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:46.766419 28200 raft_consensus.cc:738] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:46.767695 28200 consensus_queue.cc:260] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 19:58:46.773605 27999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f356a2522e646889c023d4b3d26f9cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.774519 28143 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 361ffd5c9a994e0397e6748827466d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.774847 27992 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62ccdb07048e4762b1b31c36b14610f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.778848 28145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f356a2522e646889c023d4b3d26f9cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.779783 27994 tablet_service.cc:1468] Processing CreateTablet for tablet 456d9fe0918b423abeab5c39372bcbe9 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250905 19:58:46.781257 27994 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 456d9fe0918b423abeab5c39372bcbe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.781008 28141 tablet_service.cc:1468] Processing CreateTablet for tablet 18ed409dabfc45b1843e7d8a316a047d (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250905 19:58:46.799034 28141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18ed409dabfc45b1843e7d8a316a047d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.783605 28146 tablet_service.cc:1468] Processing CreateTablet for tablet 60054c57e2554f32a5ad379a79c2a738 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20250905 19:58:46.803390 28146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60054c57e2554f32a5ad379a79c2a738. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.806533 28070 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62ccdb07048e4762b1b31c36b14610f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.806793 28066 tablet_service.cc:1468] Processing CreateTablet for tablet 361ffd5c9a994e0397e6748827466d3d (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250905 19:58:46.808163 28066 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 361ffd5c9a994e0397e6748827466d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.789417 27996 tablet_service.cc:1468] Processing CreateTablet for tablet 2bd6bb972f14472a89ccf99de880efc2 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250905 19:58:46.811292 27996 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bd6bb972f14472a89ccf99de880efc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.811125 27998 tablet_service.cc:1468] Processing CreateTablet for tablet 18ed409dabfc45b1843e7d8a316a047d (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250905 19:58:46.789292 28071 tablet_service.cc:1468] Processing CreateTablet for tablet fc6a8d5d253643cea8582bef1c4173d6 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20250905 19:58:46.812494 27998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18ed409dabfc45b1843e7d8a316a047d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.813030 28071 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc6a8d5d253643cea8582bef1c4173d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.792959 28073 tablet_service.cc:1468] Processing CreateTablet for tablet b13af7609cd04cf5b21745527b70074d (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250905 19:58:46.816119 28073 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b13af7609cd04cf5b21745527b70074d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.794696 28202 tablet_bootstrap.cc:492] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:46.818835 28202 ts_tablet_manager.cc:1397] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.142s user 0.013s sys 0.000s
I20250905 19:58:46.794821 28148 tablet_service.cc:1468] Processing CreateTablet for tablet 18c49d7ed5d64445afa2c6ee7a2e27d1 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250905 19:58:46.820031 27995 tablet_service.cc:1468] Processing CreateTablet for tablet 60054c57e2554f32a5ad379a79c2a738 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20250905 19:58:46.820798 28148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18c49d7ed5d64445afa2c6ee7a2e27d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.821719 27995 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60054c57e2554f32a5ad379a79c2a738. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.797433 28201 raft_consensus.cc:357] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:46.822930 28201 raft_consensus.cc:383] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:46.823282 28201 raft_consensus.cc:738] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:46.824529 28201 consensus_queue.cc:260] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:46.827361 28202 raft_consensus.cc:357] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:46.844301 27997 tablet_service.cc:1468] Processing CreateTablet for tablet 18c49d7ed5d64445afa2c6ee7a2e27d1 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250905 19:58:46.845750 27997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18c49d7ed5d64445afa2c6ee7a2e27d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.848248 28035 heartbeater.cc:499] Master 127.26.58.190:38969 was elected leader, sending a full tablet report...
I20250905 19:58:46.835158 28202 raft_consensus.cc:383] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:46.857402 28202 raft_consensus.cc:738] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:46.863761 28202 consensus_queue.cc:260] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:46.864898 28068 tablet_service.cc:1468] Processing CreateTablet for tablet 456d9fe0918b423abeab5c39372bcbe9 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250905 19:58:46.867461 28072 tablet_service.cc:1468] Processing CreateTablet for tablet 2f356a2522e646889c023d4b3d26f9cc (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250905 19:58:46.872879 27997 tablet_service.cc:1468] Processing CreateTablet for tablet aace14abd237420c8211ce611ea09f0e (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250905 19:58:46.876757 28149 tablet_service.cc:1468] Processing CreateTablet for tablet aace14abd237420c8211ce611ea09f0e (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250905 19:58:46.878777 28140 tablet_service.cc:1468] Processing CreateTablet for tablet 5f31007f29044bd1817fcf19ba904ad3 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250905 19:58:46.866288 28068 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 456d9fe0918b423abeab5c39372bcbe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.916122 28072 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f356a2522e646889c023d4b3d26f9cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.916734 28184 heartbeater.cc:499] Master 127.26.58.190:38969 was elected leader, sending a full tablet report...
I20250905 19:58:46.912122 28149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aace14abd237420c8211ce611ea09f0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.926823 28140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f31007f29044bd1817fcf19ba904ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.931289 28075 tablet_service.cc:1468] Processing CreateTablet for tablet 2bd6bb972f14472a89ccf99de880efc2 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250905 19:58:46.932977 28201 ts_tablet_manager.cc:1428] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.175s user 0.006s sys 0.069s
I20250905 19:58:46.915377 27997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aace14abd237420c8211ce611ea09f0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.934259 28110 heartbeater.cc:499] Master 127.26.58.190:38969 was elected leader, sending a full tablet report...
I20250905 19:58:46.934739 28075 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bd6bb972f14472a89ccf99de880efc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.934221 28201 tablet_bootstrap.cc:492] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:46.943076 27991 tablet_service.cc:1468] Processing CreateTablet for tablet 5f31007f29044bd1817fcf19ba904ad3 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250905 19:58:46.929127 28144 tablet_service.cc:1468] Processing CreateTablet for tablet d06337b10d154efbbaaf3c152d01f6dd (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20250905 19:58:46.944096 28000 tablet_service.cc:1468] Processing CreateTablet for tablet d06337b10d154efbbaaf3c152d01f6dd (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20250905 19:58:46.944509 27991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f31007f29044bd1817fcf19ba904ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.945396 28000 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d06337b10d154efbbaaf3c152d01f6dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.945359 28144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d06337b10d154efbbaaf3c152d01f6dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.965143 28146 tablet_service.cc:1468] Processing CreateTablet for tablet 1f65256308814d3a9a4a01db1b910611 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250905 19:58:46.974748 28146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f65256308814d3a9a4a01db1b910611. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.989797 28075 tablet_service.cc:1468] Processing CreateTablet for tablet 60054c57e2554f32a5ad379a79c2a738 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20250905 19:58:46.997792 28075 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60054c57e2554f32a5ad379a79c2a738. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.003546 28200 ts_tablet_manager.cc:1428] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.318s user 0.014s sys 0.032s
I20250905 19:58:46.998167 28144 tablet_service.cc:1468] Processing CreateTablet for tablet 9a9976ebf1c1415a85592e195bad9f36 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
W20250905 19:58:47.015275 28185 tablet.cc:2378] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 19:58:47.023643 27993 tablet_service.cc:1468] Processing CreateTablet for tablet 9a9976ebf1c1415a85592e195bad9f36 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20250905 19:58:47.025282 27993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a9976ebf1c1415a85592e195bad9f36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.024153 28201 tablet_bootstrap.cc:654] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:47.025492 28074 tablet_service.cc:1468] Processing CreateTablet for tablet 18ed409dabfc45b1843e7d8a316a047d (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250905 19:58:47.030711 28074 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18ed409dabfc45b1843e7d8a316a047d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:46.958142 27992 tablet_service.cc:1468] Processing CreateTablet for tablet 1f65256308814d3a9a4a01db1b910611 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250905 19:58:47.048486 28070 tablet_service.cc:1468] Processing CreateTablet for tablet d06337b10d154efbbaaf3c152d01f6dd (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20250905 19:58:47.049378 27994 tablet_service.cc:1468] Processing CreateTablet for tablet 8e0ac2dfe59d423189dd8d72eee98a42 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250905 19:58:47.050117 28070 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d06337b10d154efbbaaf3c152d01f6dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.046056 28069 tablet_service.cc:1468] Processing CreateTablet for tablet aace14abd237420c8211ce611ea09f0e (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250905 19:58:47.062775 27992 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f65256308814d3a9a4a01db1b910611. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.064360 27994 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e0ac2dfe59d423189dd8d72eee98a42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.058049 28067 tablet_service.cc:1468] Processing CreateTablet for tablet 5f31007f29044bd1817fcf19ba904ad3 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250905 19:58:47.066895 28071 tablet_service.cc:1468] Processing CreateTablet for tablet 18c49d7ed5d64445afa2c6ee7a2e27d1 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250905 19:58:47.068768 27998 tablet_service.cc:1468] Processing CreateTablet for tablet 7eae4eae1373464d85793607943a7c4d (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20250905 19:58:47.070036 27998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eae4eae1373464d85793607943a7c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.077190 27995 tablet_service.cc:1468] Processing CreateTablet for tablet 6db33851e9914ca58560e8902701dca3 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250905 19:58:47.089685 28200 tablet_bootstrap.cc:492] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:47.095360 27996 tablet_service.cc:1468] Processing CreateTablet for tablet 1d1a8f2db2604a879529dda40b457a4a (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20250905 19:58:47.096000 28147 tablet_service.cc:1468] Processing CreateTablet for tablet 7eae4eae1373464d85793607943a7c4d (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20250905 19:58:47.097378 28147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eae4eae1373464d85793607943a7c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.099972 28145 tablet_service.cc:1468] Processing CreateTablet for tablet 8e0ac2dfe59d423189dd8d72eee98a42 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250905 19:58:47.101344 28145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e0ac2dfe59d423189dd8d72eee98a42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.103801 28066 tablet_service.cc:1468] Processing CreateTablet for tablet 1f65256308814d3a9a4a01db1b910611 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250905 19:58:47.103999 28142 tablet_service.cc:1468] Processing CreateTablet for tablet 6db33851e9914ca58560e8902701dca3 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250905 19:58:47.105857 28207 raft_consensus.cc:491] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:47.106483 28207 raft_consensus.cc:513] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:47.110622 28209 raft_consensus.cc:491] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:47.114988 28209 raft_consensus.cc:513] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:47.111331 27995 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6db33851e9914ca58560e8902701dca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.115816 28067 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f31007f29044bd1817fcf19ba904ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.140377 28143 tablet_service.cc:1468] Processing CreateTablet for tablet 1d1a8f2db2604a879529dda40b457a4a (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20250905 19:58:47.141971 28202 ts_tablet_manager.cc:1428] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.323s user 0.021s sys 0.039s
I20250905 19:58:47.142596 28201 tablet_bootstrap.cc:492] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:47.142997 28144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a9976ebf1c1415a85592e195bad9f36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.144383 28068 tablet_service.cc:1468] Processing CreateTablet for tablet 9a9976ebf1c1415a85592e195bad9f36 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20250905 19:58:47.146147 28073 tablet_service.cc:1468] Processing CreateTablet for tablet 8e0ac2dfe59d423189dd8d72eee98a42 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250905 19:58:47.149646 28200 tablet_bootstrap.cc:654] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:47.152328 28148 tablet_service.cc:1468] Processing CreateTablet for tablet b12e279a4fab4ea3a15dadc55d224ddb (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250905 19:58:47.149670 28071 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18c49d7ed5d64445afa2c6ee7a2e27d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.111018 28207 leader_election.cc:290] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:47.152462 28209 leader_election.cc:290] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:47.168677 27996 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d1a8f2db2604a879529dda40b457a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.162171 28201 ts_tablet_manager.cc:1397] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.228s user 0.017s sys 0.013s
I20250905 19:58:47.180680 28142 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6db33851e9914ca58560e8902701dca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.180080 28069 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aace14abd237420c8211ce611ea09f0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.193113 28072 tablet_service.cc:1468] Processing CreateTablet for tablet 7eae4eae1373464d85793607943a7c4d (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20250905 19:58:47.193421 27999 tablet_service.cc:1468] Processing CreateTablet for tablet 5ed920d4c0144d62acf66f595bbf34fd (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250905 19:58:47.193471 28141 tablet_service.cc:1468] Processing CreateTablet for tablet 24cf07baf43247b6bf63063106b79836 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250905 19:58:47.194902 28202 tablet_bootstrap.cc:492] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:47.196260 28146 tablet_service.cc:1468] Processing CreateTablet for tablet 5ed920d4c0144d62acf66f595bbf34fd (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250905 19:58:47.194918 27999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ed920d4c0144d62acf66f595bbf34fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.200939 27991 tablet_service.cc:1468] Processing CreateTablet for tablet b12e279a4fab4ea3a15dadc55d224ddb (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250905 19:58:47.213639 28072 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eae4eae1373464d85793607943a7c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.183193 28201 raft_consensus.cc:357] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:47.220029 28201 raft_consensus.cc:383] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:47.219929 28074 tablet_service.cc:1468] Processing CreateTablet for tablet 6db33851e9914ca58560e8902701dca3 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250905 19:58:47.220487 28201 raft_consensus.cc:738] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:47.221480 28074 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6db33851e9914ca58560e8902701dca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.221379 28201 consensus_queue.cc:260] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:47.233637 28066 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f65256308814d3a9a4a01db1b910611. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.234774 28201 ts_tablet_manager.cc:1428] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.054s user 0.006s sys 0.002s
I20250905 19:58:47.211196 28143 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d1a8f2db2604a879529dda40b457a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.216826 28206 raft_consensus.cc:491] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:47.260736 28206 raft_consensus.cc:513] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 19:58:47.262459 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:47.243803 28149 tablet_service.cc:1468] Processing CreateTablet for tablet 05b2fd124328472aa3e3f62e0b327582 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250905 19:58:47.247818 28068 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a9976ebf1c1415a85592e195bad9f36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.257666 28075 tablet_service.cc:1468] Processing CreateTablet for tablet 1d1a8f2db2604a879529dda40b457a4a (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20250905 19:58:47.279717 28201 tablet_bootstrap.cc:492] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:47.282418 28206 leader_election.cc:290] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:47.294967 28141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24cf07baf43247b6bf63063106b79836. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.320078 28200 tablet_bootstrap.cc:492] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:47.320844 28200 ts_tablet_manager.cc:1397] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.231s user 0.012s sys 0.004s
I20250905 19:58:47.323881 28202 tablet_bootstrap.cc:654] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:47.327275 28149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05b2fd124328472aa3e3f62e0b327582. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.329389 28148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b12e279a4fab4ea3a15dadc55d224ddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.334784 28146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ed920d4c0144d62acf66f595bbf34fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.312513 27997 tablet_service.cc:1468] Processing CreateTablet for tablet 24cf07baf43247b6bf63063106b79836 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250905 19:58:47.341929 28207 raft_consensus.cc:491] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:47.341750 27993 tablet_service.cc:1468] Processing CreateTablet for tablet fd2b679c53994263846f51ceec97a150 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250905 19:58:47.342574 28207 raft_consensus.cc:513] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:47.344554 28207 leader_election.cc:290] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:47.345979 27992 tablet_service.cc:1468] Processing CreateTablet for tablet 05b2fd124328472aa3e3f62e0b327582 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250905 19:58:47.349233 28140 tablet_service.cc:1468] Processing CreateTablet for tablet fd2b679c53994263846f51ceec97a150 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250905 19:58:47.311439 28073 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e0ac2dfe59d423189dd8d72eee98a42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.379889 28070 tablet_service.cc:1468] Processing CreateTablet for tablet 5ed920d4c0144d62acf66f595bbf34fd (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250905 19:58:47.382227 28070 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ed920d4c0144d62acf66f595bbf34fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.390892 28075 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d1a8f2db2604a879529dda40b457a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.395231 28140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd2b679c53994263846f51ceec97a150. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.369222 28200 raft_consensus.cc:357] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:47.303279 27991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b12e279a4fab4ea3a15dadc55d224ddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.398166 28200 raft_consensus.cc:383] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:47.404672 28200 raft_consensus.cc:738] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:47.405704 28200 consensus_queue.cc:260] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:47.409515 28200 ts_tablet_manager.cc:1428] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.088s user 0.006s sys 0.005s
I20250905 19:58:47.410842 27993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd2b679c53994263846f51ceec97a150. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.412005 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:47.425966 27997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24cf07baf43247b6bf63063106b79836. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.431129 27992 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05b2fd124328472aa3e3f62e0b327582. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:58:47.413260 27973 leader_election.cc:343] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:47.414179 28147 tablet_service.cc:1468] Processing CreateTablet for tablet 0860997b3256400084f5884190bbe6e2 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20250905 19:58:47.419355 27999 tablet_service.cc:1468] Processing CreateTablet for tablet 10df9d019f2e4e208f4e23d50f551ad6 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250905 19:58:47.420473 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:47.433488 27994 tablet_service.cc:1468] Processing CreateTablet for tablet 0860997b3256400084f5884190bbe6e2 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
W20250905 19:58:47.434834 28122 leader_election.cc:343] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:58:47.435753 27973 leader_election.cc:343] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:47.437322 28072 tablet_service.cc:1468] Processing CreateTablet for tablet 24cf07baf43247b6bf63063106b79836 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250905 19:58:47.438236 27998 tablet_service.cc:1468] Processing CreateTablet for tablet ef7f42a26b64466a9d6b1eae6df0e614 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20250905 19:58:47.422034 28067 tablet_service.cc:1468] Processing CreateTablet for tablet b12e279a4fab4ea3a15dadc55d224ddb (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250905 19:58:47.456045 28201 tablet_bootstrap.cc:654] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:47.446586 27999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10df9d019f2e4e208f4e23d50f551ad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.460773 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:47.461534 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:47.461453 27996 tablet_service.cc:1468] Processing CreateTablet for tablet c0889204295e4daa99d81e5b2a45ac1c (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250905 19:58:47.462456 28072 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24cf07baf43247b6bf63063106b79836. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.462245 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:47.475019 28159 raft_consensus.cc:2466] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 0.
W20250905 19:58:47.475562 27975 leader_election.cc:343] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:47.476040 27975 leader_election.cc:304] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:47.477038 28207 raft_consensus.cc:2747] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:47.465888 28147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0860997b3256400084f5884190bbe6e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.478529 27975 leader_election.cc:304] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:47.479430 28207 raft_consensus.cc:2802] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:47.479797 28207 raft_consensus.cc:491] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:47.480130 28207 raft_consensus.cc:3058] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:47.468911 28202 tablet_bootstrap.cc:492] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:47.482918 28202 ts_tablet_manager.cc:1397] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.288s user 0.011s sys 0.007s
I20250905 19:58:47.484365 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:47.470729 27998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef7f42a26b64466a9d6b1eae6df0e614. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.467881 28145 tablet_service.cc:1468] Processing CreateTablet for tablet ef7f42a26b64466a9d6b1eae6df0e614 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
W20250905 19:58:47.477020 28050 leader_election.cc:343] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:58:47.490216 28051 leader_election.cc:343] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:47.493003 28051 leader_election.cc:304] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:47.493531 28200 tablet_bootstrap.cc:492] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:47.493872 28209 raft_consensus.cc:2747] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:47.494724 27996 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0889204295e4daa99d81e5b2a45ac1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.494993 28145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef7f42a26b64466a9d6b1eae6df0e614. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.493026 28067 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b12e279a4fab4ea3a15dadc55d224ddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.499428 28144 tablet_service.cc:1468] Processing CreateTablet for tablet 10df9d019f2e4e208f4e23d50f551ad6 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250905 19:58:47.485960 28202 raft_consensus.cc:357] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:47.500753 28144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10df9d019f2e4e208f4e23d50f551ad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.500874 28202 raft_consensus.cc:383] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:47.501250 28202 raft_consensus.cc:738] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:47.445484 28000 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54144 (request call id 27) took 1014 ms. Trace:
I20250905 19:58:47.502686 28000 rpcz_store.cc:276] 0905 19:58:46.431190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:58:46.943163 (+511973us) service_pool.cc:225] Handling call
0905 19:58:46.944943 (+ 1780us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0905 19:58:46.945045 (+ 102us) ts_tablet_manager.cc:676] Creating new metadata...
0905 19:58:47.331760 (+386715us) tablet_metadata.cc:641] Metadata flushed
0905 19:58:47.436801 (+105041us) tablet_replica.cc:206] Creating consensus instance
0905 19:58:47.445407 (+ 8606us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:58:47.505086 28000 tablet_service.cc:1468] Processing CreateTablet for tablet 0e4ec392d0d54040b0b2162eb0601c0d (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20250905 19:58:47.510771 28202 consensus_queue.cc:260] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:47.519850 28069 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59536 (request call id 26) took 1073 ms. Trace:
I20250905 19:58:47.520454 28069 rpcz_store.cc:276] 0905 19:58:46.446882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:58:47.045059 (+598177us) service_pool.cc:225] Handling call
0905 19:58:47.114820 (+ 69761us) spinlock_profiling.cc:243] Waited 2.82 ms on lock 0x7b5800204f98. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac671cc 00007fe49ac66d8b 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2beb571 00007fe4a2bc8903 00007fe4a2b79219 00007fe49c121bf4 00007fe49c121b82 00007fe49c121ab1
0905 19:58:47.156381 (+ 41561us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0905 19:58:47.161027 (+ 4646us) ts_tablet_manager.cc:676] Creating new metadata...
0905 19:58:47.383731 (+222704us) tablet_metadata.cc:641] Metadata flushed
0905 19:58:47.519187 (+135456us) tablet_replica.cc:206] Creating consensus instance
0905 19:58:47.519757 (+ 570us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6491008}
I20250905 19:58:47.502206 27995 tablet_service.cc:1468] Processing CreateTablet for tablet 7ca47360fb0f4be6a95a798d0a6d25b8 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250905 19:58:47.548878 28142 tablet_service.cc:1468] Processing CreateTablet for tablet c0889204295e4daa99d81e5b2a45ac1c (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250905 19:58:47.549593 27997 tablet_service.cc:1468] Processing CreateTablet for tablet 1b18b7149ff3439a8dde09149cf336b3 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20250905 19:58:47.525233 28071 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59536 (request call id 25) took 1078 ms. Trace:
I20250905 19:58:47.527680 28143 tablet_service.cc:1468] Processing CreateTablet for tablet 7ca47360fb0f4be6a95a798d0a6d25b8 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250905 19:58:47.534888 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:47.550551 28071 rpcz_store.cc:276] 0905 19:58:46.446544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:58:47.034179 (+587635us) service_pool.cc:225] Handling call
0905 19:58:47.108053 (+ 73874us) spinlock_profiling.cc:243] Waited 16.9 ms on lock 0x7b5800204f98. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2beb571 00007fe4a2bc8903 00007fe4a2b79219 00007fe49c121bf4 00007fe49c121b82 00007fe49c121ab1
0905 19:58:47.127005 (+ 18952us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0905 19:58:47.136019 (+ 9014us) ts_tablet_manager.cc:676] Creating new metadata...
0905 19:58:47.376558 (+240539us) tablet_metadata.cc:641] Metadata flushed
0905 19:58:47.524644 (+148086us) tablet_replica.cc:206] Creating consensus instance
0905 19:58:47.525162 (+ 518us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":38787840}
W20250905 19:58:47.552188 28125 leader_election.cc:343] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:47.552665 28125 leader_election.cc:304] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:47.553367 28071 tablet_service.cc:1468] Processing CreateTablet for tablet fd2b679c53994263846f51ceec97a150 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250905 19:58:47.553833 28206 raft_consensus.cc:2747] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:47.543300 28147 tablet_service.cc:1468] Processing CreateTablet for tablet 0e4ec392d0d54040b0b2162eb0601c0d (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20250905 19:58:47.555599 28071 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd2b679c53994263846f51ceec97a150. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.555274 28074 tablet_service.cc:1468] Processing CreateTablet for tablet 10df9d019f2e4e208f4e23d50f551ad6 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250905 19:58:47.556520 28074 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10df9d019f2e4e208f4e23d50f551ad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.532743 28207 raft_consensus.cc:513] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:47.533660 28000 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e4ec392d0d54040b0b2162eb0601c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.533136 28066 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59536 (request call id 28) took 1085 ms. Trace:
I20250905 19:58:47.559183 27994 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0860997b3256400084f5884190bbe6e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.558895 28066 rpcz_store.cc:276] 0905 19:58:46.447649 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:58:47.102740 (+655091us) service_pool.cc:225] Handling call
0905 19:58:47.137863 (+ 35123us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0905 19:58:47.153575 (+ 15712us) ts_tablet_manager.cc:676] Creating new metadata...
0905 19:58:47.423932 (+270357us) tablet_metadata.cc:641] Metadata flushed
0905 19:58:47.532540 (+108608us) tablet_replica.cc:206] Creating consensus instance
0905 19:58:47.533068 (+ 528us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:58:47.561498 28200 tablet_bootstrap.cc:654] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:47.561887 28066 tablet_service.cc:1468] Processing CreateTablet for tablet 0860997b3256400084f5884190bbe6e2 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20250905 19:58:47.563215 28066 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0860997b3256400084f5884190bbe6e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.563253 28202 ts_tablet_manager.cc:1428] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.080s user 0.006s sys 0.002s
I20250905 19:58:47.566397 28141 tablet_service.cc:1468] Processing CreateTablet for tablet 1b18b7149ff3439a8dde09149cf336b3 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20250905 19:58:47.567431 27995 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ca47360fb0f4be6a95a798d0a6d25b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.600631 27997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b18b7149ff3439a8dde09149cf336b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.570845 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:58:47.601235 28084 raft_consensus.cc:3058] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:47.570935 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:47.571694 27991 tablet_service.cc:1468] Processing CreateTablet for tablet 5c21c3755f90405db03f98780efdb0b5 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250905 19:58:47.580757 28068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59536 (request call id 29) took 1132 ms. Trace:
I20250905 19:58:47.549809 28069 tablet_service.cc:1468] Processing CreateTablet for tablet 05b2fd124328472aa3e3f62e0b327582 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250905 19:58:47.605602 28159 raft_consensus.cc:3058] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:47.581645 28207 leader_election.cc:290] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:47.583186 28146 tablet_service.cc:1468] Processing CreateTablet for tablet 5c21c3755f90405db03f98780efdb0b5 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250905 19:58:47.550398 28142 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0889204295e4daa99d81e5b2a45ac1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.595753 27999 tablet_service.cc:1468] Processing CreateTablet for tablet e802ce1da002477b8896acf7bf7c84a7 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250905 19:58:47.590109 28073 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59536 (request call id 30) took 1066 ms. Trace:
I20250905 19:58:47.568444 28202 tablet_bootstrap.cc:492] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:47.604292 28068 rpcz_store.cc:276] 0905 19:58:46.447992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:58:47.143426 (+695434us) service_pool.cc:225] Handling call
0905 19:58:47.175510 (+ 32084us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0905 19:58:47.176813 (+ 1303us) ts_tablet_manager.cc:676] Creating new metadata...
0905 19:58:47.416837 (+240024us) tablet_metadata.cc:641] Metadata flushed
0905 19:58:47.580094 (+163257us) tablet_replica.cc:206] Creating consensus instance
0905 19:58:47.580654 (+ 560us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:58:47.608595 28069 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05b2fd124328472aa3e3f62e0b327582. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.611208 28148 tablet_service.cc:1468] Processing CreateTablet for tablet e802ce1da002477b8896acf7bf7c84a7 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250905 19:58:47.609578 28146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c21c3755f90405db03f98780efdb0b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.617170 27991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c21c3755f90405db03f98780efdb0b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.615000 28073 rpcz_store.cc:276] 0905 19:58:46.523581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:58:47.145167 (+621586us) service_pool.cc:225] Handling call
0905 19:58:47.246559 (+101392us) spinlock_profiling.cc:243] Waited 2.99 ms on lock 0x7b5800204f98. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac674d2 00007fe4a39a4b39 00007fe4a2b07eb8 00007fe4a2bc89c0 00007fe4a2b79219 00007fe49c121bf4 00007fe49c121b82 00007fe49c121ab1 00007fe49c121a2c
0905 19:58:47.246626 (+ 67us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0905 19:58:47.246763 (+ 137us) ts_tablet_manager.cc:676] Creating new metadata...
0905 19:58:47.525571 (+278808us) tablet_metadata.cc:641] Metadata flushed
0905 19:58:47.589546 (+ 63975us) tablet_replica.cc:206] Creating consensus instance
0905 19:58:47.590038 (+ 492us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6884224}
I20250905 19:58:47.623636 27992 tablet_service.cc:1468] Processing CreateTablet for tablet a8574207dbad48be8dcd599e9cae1ded (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250905 19:58:47.614246 28084 raft_consensus.cc:2466] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:58:47.622366 28202 tablet_bootstrap.cc:654] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:47.624182 28149 tablet_service.cc:1468] Processing CreateTablet for tablet 4f2279535926437ba84ae91f2b83b3d3 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20250905 19:58:47.629555 27973 leader_election.cc:304] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:47.630581 28207 raft_consensus.cc:2802] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:47.631076 28201 tablet_bootstrap.cc:492] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:47.637938 28068 tablet_service.cc:1468] Processing CreateTablet for tablet 7ca47360fb0f4be6a95a798d0a6d25b8 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250905 19:58:47.639361 28068 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ca47360fb0f4be6a95a798d0a6d25b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.625969 28073 tablet_service.cc:1468] Processing CreateTablet for tablet c0889204295e4daa99d81e5b2a45ac1c (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250905 19:58:47.644846 27992 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8574207dbad48be8dcd599e9cae1ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.645046 28147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e4ec392d0d54040b0b2162eb0601c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.643712 28201 ts_tablet_manager.cc:1397] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.364s user 0.033s sys 0.013s
I20250905 19:58:47.653666 28143 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ca47360fb0f4be6a95a798d0a6d25b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.655251 28149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f2279535926437ba84ae91f2b83b3d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.658149 28148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e802ce1da002477b8896acf7bf7c84a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.665668 28070 tablet_service.cc:1468] Processing CreateTablet for tablet 0e4ec392d0d54040b0b2162eb0601c0d (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20250905 19:58:47.667294 28200 tablet_bootstrap.cc:492] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:47.667851 28200 ts_tablet_manager.cc:1397] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.175s user 0.016s sys 0.001s
I20250905 19:58:47.663250 28140 tablet_service.cc:1468] Processing CreateTablet for tablet a8574207dbad48be8dcd599e9cae1ded (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250905 19:58:47.671173 28200 raft_consensus.cc:357] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:47.672142 28200 raft_consensus.cc:383] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:47.672475 28200 raft_consensus.cc:738] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
W20250905 19:58:47.673362 27956 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250905 19:58:47.673292 28200 consensus_queue.cc:260] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:47.663854 28207 raft_consensus.cc:695] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:58:47.675086 28073 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0889204295e4daa99d81e5b2a45ac1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.675436 28207 consensus_queue.cc:237] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:47.677978 28200 ts_tablet_manager.cc:1428] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.010s user 0.001s sys 0.007s
I20250905 19:58:47.679818 27996 tablet_service.cc:1468] Processing CreateTablet for tablet 8a8d8393352b4214a20c1683852d4fdd (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20250905 19:58:47.686942 27999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e802ce1da002477b8896acf7bf7c84a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.691936 28201 raft_consensus.cc:357] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:47.692790 28201 raft_consensus.cc:383] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:47.693121 28201 raft_consensus.cc:738] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:47.693890 28201 consensus_queue.cc:260] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:47.694748 28141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b18b7149ff3439a8dde09149cf336b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.695281 28070 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e4ec392d0d54040b0b2162eb0601c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.700387 28159 raft_consensus.cc:2466] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:58:47.703027 28067 tablet_service.cc:1468] Processing CreateTablet for tablet 1b18b7149ff3439a8dde09149cf336b3 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20250905 19:58:47.706820 27996 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a8d8393352b4214a20c1683852d4fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.707041 28075 tablet_service.cc:1468] Processing CreateTablet for tablet ef7f42a26b64466a9d6b1eae6df0e614 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20250905 19:58:47.707856 27998 tablet_service.cc:1468] Processing CreateTablet for tablet a3efc6666ba5411c9b50c9d6e34089ad (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20250905 19:58:47.711189 28140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8574207dbad48be8dcd599e9cae1ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.712554 28145 tablet_service.cc:1468] Processing CreateTablet for tablet 8a8d8393352b4214a20c1683852d4fdd (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20250905 19:58:47.714998 27993 tablet_service.cc:1468] Processing CreateTablet for tablet 4f2279535926437ba84ae91f2b83b3d3 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20250905 19:58:47.717469 28200 tablet_bootstrap.cc:492] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:47.720430 28206 raft_consensus.cc:491] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:47.754161 28206 raft_consensus.cc:513] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:47.756405 28206 leader_election.cc:290] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:47.731424 28144 tablet_service.cc:1468] Processing CreateTablet for tablet a042aed9ffcb49c19e07c53c69bd1b95 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20250905 19:58:47.740597 28201 ts_tablet_manager.cc:1428] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.087s user 0.006s sys 0.000s
I20250905 19:58:47.743119 28067 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b18b7149ff3439a8dde09149cf336b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.757565 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:47.742027 28142 tablet_service.cc:1468] Processing CreateTablet for tablet a3efc6666ba5411c9b50c9d6e34089ad (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
W20250905 19:58:47.759218 28122 leader_election.cc:343] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:47.760313 27993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f2279535926437ba84ae91f2b83b3d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.760547 27908 catalog_manager.cc:5582] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:47.734069 28072 tablet_service.cc:1468] Processing CreateTablet for tablet 5c21c3755f90405db03f98780efdb0b5 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250905 19:58:47.763257 28072 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c21c3755f90405db03f98780efdb0b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.765954 28201 tablet_bootstrap.cc:492] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:47.757368 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:47.769009 27998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3efc6666ba5411c9b50c9d6e34089ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.768954 28143 tablet_service.cc:1468] Processing CreateTablet for tablet 4414c08fe910403b80af2d558aac36b2 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20250905 19:58:47.769404 28010 raft_consensus.cc:2466] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 0.
I20250905 19:58:47.753542 28145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a8d8393352b4214a20c1683852d4fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.770641 28125 leader_election.cc:304] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:47.771798 28206 raft_consensus.cc:2802] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:47.772161 28206 raft_consensus.cc:491] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:47.772486 28206 raft_consensus.cc:3058] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:47.773969 27995 tablet_service.cc:1468] Processing CreateTablet for tablet a042aed9ffcb49c19e07c53c69bd1b95 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20250905 19:58:47.783296 27995 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a042aed9ffcb49c19e07c53c69bd1b95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.795019 28071 tablet_service.cc:1468] Processing CreateTablet for tablet e802ce1da002477b8896acf7bf7c84a7 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250905 19:58:47.795506 28206 raft_consensus.cc:513] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:47.796672 28071 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e802ce1da002477b8896acf7bf7c84a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.798770 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:47.799090 28075 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef7f42a26b64466a9d6b1eae6df0e614. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.799461 28010 raft_consensus.cc:3058] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:47.791081 28149 tablet_service.cc:1468] Processing CreateTablet for tablet a002b516fab044b998f136c341a88b74 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250905 19:58:47.801285 28149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a002b516fab044b998f136c341a88b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.804069 28147 tablet_service.cc:1468] Processing CreateTablet for tablet fba42101cd094077aa85e3f9186366e7 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20250905 19:58:47.805418 28147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fba42101cd094077aa85e3f9186366e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.810562 28074 tablet_service.cc:1468] Processing CreateTablet for tablet 4f2279535926437ba84ae91f2b83b3d3 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20250905 19:58:47.811933 28074 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f2279535926437ba84ae91f2b83b3d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.815728 28070 tablet_service.cc:1468] Processing CreateTablet for tablet a8574207dbad48be8dcd599e9cae1ded (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250905 19:58:47.793956 28146 tablet_service.cc:1468] Processing CreateTablet for tablet 10dd790c7b5c4cc5978c3545b5a06868 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250905 19:58:47.827101 27992 tablet_service.cc:1468] Processing CreateTablet for tablet a002b516fab044b998f136c341a88b74 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250905 19:58:47.833112 27994 tablet_service.cc:1468] Processing CreateTablet for tablet 4414c08fe910403b80af2d558aac36b2 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20250905 19:58:47.830349 28201 tablet_bootstrap.cc:654] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:47.834189 27992 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a002b516fab044b998f136c341a88b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.841655 28148 tablet_service.cc:1468] Processing CreateTablet for tablet 50ac9970ddf042fda15f38b623aa7dd0 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20250905 19:58:47.845697 28010 raft_consensus.cc:2466] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 19:58:47.847265 28125 leader_election.cc:304] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:58:47.845408 27996 tablet_service.cc:1468] Processing CreateTablet for tablet fba42101cd094077aa85e3f9186366e7 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20250905 19:58:47.852703 28144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a042aed9ffcb49c19e07c53c69bd1b95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.853505 28142 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3efc6666ba5411c9b50c9d6e34089ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.855459 28148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50ac9970ddf042fda15f38b623aa7dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.858816 28143 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4414c08fe910403b80af2d558aac36b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.865471 27997 tablet_service.cc:1468] Processing CreateTablet for tablet 50ac9970ddf042fda15f38b623aa7dd0 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20250905 19:58:47.876523 27991 tablet_service.cc:1468] Processing CreateTablet for tablet ff701570f51b481a81cb71ede12a494f (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250905 19:58:47.877529 28200 tablet_bootstrap.cc:654] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:47.881278 27994 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4414c08fe910403b80af2d558aac36b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.883739 28145 tablet_service.cc:1468] Processing CreateTablet for tablet ff701570f51b481a81cb71ede12a494f (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250905 19:58:47.887745 28146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10dd790c7b5c4cc5978c3545b5a06868. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.835985 28206 leader_election.cc:290] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:47.910676 28070 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8574207dbad48be8dcd599e9cae1ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.910864 28069 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59536 (request call id 37) took 1102 ms. Trace:
I20250905 19:58:47.911273 28069 rpcz_store.cc:276] 0905 19:58:46.808140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:58:47.548357 (+740217us) service_pool.cc:225] Handling call
0905 19:58:47.605712 (+ 57355us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0905 19:58:47.605871 (+ 159us) ts_tablet_manager.cc:676] Creating new metadata...
0905 19:58:47.775047 (+169176us) tablet_metadata.cc:641] Metadata flushed
0905 19:58:47.836507 (+ 61460us) tablet_replica.cc:206] Creating consensus instance
0905 19:58:47.910794 (+ 74287us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:58:47.913295 28069 tablet_service.cc:1468] Processing CreateTablet for tablet 8a8d8393352b4214a20c1683852d4fdd (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20250905 19:58:47.905571 27996 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fba42101cd094077aa85e3f9186366e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.920172 28068 tablet_service.cc:1468] Processing CreateTablet for tablet a3efc6666ba5411c9b50c9d6e34089ad (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20250905 19:58:47.922524 28066 tablet_service.cc:1468] Processing CreateTablet for tablet a002b516fab044b998f136c341a88b74 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250905 19:58:47.910120 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 19:58:47.926476 28122 leader_election.cc:343] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:47.926342 28073 tablet_service.cc:1468] Processing CreateTablet for tablet a042aed9ffcb49c19e07c53c69bd1b95 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20250905 19:58:47.928125 27997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50ac9970ddf042fda15f38b623aa7dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.931795 28069 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a8d8393352b4214a20c1683852d4fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.936110 28202 tablet_bootstrap.cc:492] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:47.936669 28202 ts_tablet_manager.cc:1397] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.368s user 0.023s sys 0.011s
I20250905 19:58:47.939618 28202 raft_consensus.cc:357] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:47.940389 28202 raft_consensus.cc:383] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:47.940670 28202 raft_consensus.cc:738] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:47.949101 28066 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a002b516fab044b998f136c341a88b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.918529 28206 raft_consensus.cc:2802] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:47.951051 27991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff701570f51b481a81cb71ede12a494f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.952275 28068 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3efc6666ba5411c9b50c9d6e34089ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.953060 28145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff701570f51b481a81cb71ede12a494f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.965462 28073 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a042aed9ffcb49c19e07c53c69bd1b95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.971129 28074 tablet_service.cc:1468] Processing CreateTablet for tablet 4414c08fe910403b80af2d558aac36b2 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20250905 19:58:47.958627 28202 consensus_queue.cc:260] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:47.942912 28000 tablet_service.cc:1468] Processing CreateTablet for tablet 10dd790c7b5c4cc5978c3545b5a06868 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250905 19:58:47.995503 28000 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10dd790c7b5c4cc5978c3545b5a06868. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.995903 28074 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4414c08fe910403b80af2d558aac36b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.996281 28068 tablet_service.cc:1468] Processing CreateTablet for tablet fba42101cd094077aa85e3f9186366e7 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20250905 19:58:47.998942 28068 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fba42101cd094077aa85e3f9186366e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:47.982682 28202 ts_tablet_manager.cc:1428] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.046s user 0.010s sys 0.000s
I20250905 19:58:48.016224 28202 tablet_bootstrap.cc:492] T 9f7f6e4e3a754dcda90928621168fbe7 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:47.954926 28206 raft_consensus.cc:695] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:58:48.024160 28200 tablet_bootstrap.cc:492] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:48.024734 28200 ts_tablet_manager.cc:1397] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.308s user 0.034s sys 0.040s
I20250905 19:58:48.025287 28067 tablet_service.cc:1468] Processing CreateTablet for tablet 10dd790c7b5c4cc5978c3545b5a06868 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250905 19:58:48.026728 28201 tablet_bootstrap.cc:492] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:48.027307 28201 ts_tablet_manager.cc:1397] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.262s user 0.025s sys 0.017s
I20250905 19:58:48.038830 28200 raft_consensus.cc:357] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.039711 28200 raft_consensus.cc:383] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.040053 28200 raft_consensus.cc:738] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.040849 28200 consensus_queue.cc:260] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.046633 28202 tablet_bootstrap.cc:654] T 9f7f6e4e3a754dcda90928621168fbe7 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:48.051009 28067 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10dd790c7b5c4cc5978c3545b5a06868. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:48.050662 28206 consensus_queue.cc:237] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:58:48.067006 28227 log.cc:927] Time spent T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.163s user 0.000s sys 0.001s
I20250905 19:58:48.072317 28075 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59536 (request call id 41) took 1028 ms. Trace:
I20250905 19:58:48.073017 28075 rpcz_store.cc:276] 0905 19:58:47.043704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:58:47.616447 (+572743us) service_pool.cc:225] Handling call
0905 19:58:47.798554 (+182107us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0905 19:58:47.798675 (+ 121us) ts_tablet_manager.cc:676] Creating new metadata...
0905 19:58:47.885318 (+ 86643us) tablet_metadata.cc:641] Metadata flushed
0905 19:58:48.071698 (+186380us) tablet_replica.cc:206] Creating consensus instance
0905 19:58:48.072240 (+ 542us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:58:48.074692 28072 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59536 (request call id 46) took 1016 ms. Trace:
I20250905 19:58:48.075131 28072 rpcz_store.cc:276] 0905 19:58:47.057817 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:58:47.733027 (+675210us) service_pool.cc:225] Handling call
0905 19:58:47.762762 (+ 29735us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0905 19:58:47.762862 (+ 100us) ts_tablet_manager.cc:676] Creating new metadata...
0905 19:58:47.801016 (+ 38154us) tablet_metadata.cc:641] Metadata flushed
0905 19:58:48.027251 (+226235us) tablet_replica.cc:206] Creating consensus instance
0905 19:58:48.074625 (+ 47374us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:58:48.077107 28072 tablet_service.cc:1468] Processing CreateTablet for tablet 50ac9970ddf042fda15f38b623aa7dd0 (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20250905 19:58:48.078347 28207 consensus_queue.cc:1035] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:48.082561 28071 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59536 (request call id 47) took 1011 ms. Trace:
I20250905 19:58:48.092671 28072 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50ac9970ddf042fda15f38b623aa7dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:48.105232 28209 raft_consensus.cc:491] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:48.083112 28071 rpcz_store.cc:276] 0905 19:58:47.070675 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:58:47.787774 (+717099us) service_pool.cc:225] Handling call
0905 19:58:47.796071 (+ 8297us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0905 19:58:47.796175 (+ 104us) ts_tablet_manager.cc:676] Creating new metadata...
0905 19:58:47.892374 (+ 96199us) tablet_metadata.cc:641] Metadata flushed
0905 19:58:48.014361 (+121987us) tablet_replica.cc:206] Creating consensus instance
0905 19:58:48.082487 (+ 68126us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:58:48.112802 28200 ts_tablet_manager.cc:1428] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.088s user 0.007s sys 0.000s
I20250905 19:58:48.112484 28075 tablet_service.cc:1468] Processing CreateTablet for tablet ff701570f51b481a81cb71ede12a494f (DEFAULT_TABLE table=test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250905 19:58:48.113812 28075 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff701570f51b481a81cb71ede12a494f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:58:48.113855 28200 tablet_bootstrap.cc:492] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:48.113996 28202 tablet_bootstrap.cc:492] T 9f7f6e4e3a754dcda90928621168fbe7 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:48.114589 28230 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:48.115090 28230 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.114595 28202 ts_tablet_manager.cc:1397] T 9f7f6e4e3a754dcda90928621168fbe7 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.099s user 0.019s sys 0.036s
I20250905 19:58:48.107482 28209 raft_consensus.cc:513] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.121779 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:48.121462 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:48.119114 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:48.120971 28202 raft_consensus.cc:357] T 9f7f6e4e3a754dcda90928621168fbe7 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.134420 28202 raft_consensus.cc:383] T 9f7f6e4e3a754dcda90928621168fbe7 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.134716 28141 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33742 (request call id 45) took 1079 ms. Trace:
I20250905 19:58:48.134855 28202 raft_consensus.cc:738] T 9f7f6e4e3a754dcda90928621168fbe7 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.135583 28202 consensus_queue.cc:260] T 9f7f6e4e3a754dcda90928621168fbe7 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.136998 28201 raft_consensus.cc:357] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.138017 28201 raft_consensus.cc:383] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250905 19:58:48.139767 28122 leader_election.cc:343] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:48.123941 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:48.149921 27906 catalog_manager.cc:5582] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
W20250905 19:58:48.151244 28050 leader_election.cc:343] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:58:48.152716 28051 leader_election.cc:343] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:48.117097 28230 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:48.154008 28051 leader_election.cc:304] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:48.119571 28209 leader_election.cc:290] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:48.135164 28141 rpcz_store.cc:276] 0905 19:58:47.055593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:58:47.565477 (+509884us) service_pool.cc:225] Handling call
0905 19:58:47.601161 (+ 35684us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0905 19:58:47.601264 (+ 103us) ts_tablet_manager.cc:676] Creating new metadata...
0905 19:58:48.014093 (+412829us) tablet_metadata.cc:641] Metadata flushed
0905 19:58:48.102888 (+ 88795us) tablet_replica.cc:206] Creating consensus instance
0905 19:58:48.134652 (+ 31764us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1825}
I20250905 19:58:48.167699 28070 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59536 (request call id 49) took 1084 ms. Trace:
I20250905 19:58:48.168144 28070 rpcz_store.cc:276] 0905 19:58:47.083616 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:58:47.814690 (+731074us) service_pool.cc:225] Handling call
0905 19:58:47.835921 (+ 21231us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0905 19:58:47.909992 (+ 74071us) ts_tablet_manager.cc:676] Creating new metadata...
0905 19:58:48.068281 (+158289us) tablet_metadata.cc:641] Metadata flushed
0905 19:58:48.167176 (+ 98895us) tablet_replica.cc:206] Creating consensus instance
0905 19:58:48.167629 (+ 453us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:58:48.169426 28209 raft_consensus.cc:2747] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:48.138384 28201 raft_consensus.cc:738] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.172128 28201 consensus_queue.cc:260] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.173216 28009 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 0.
I20250905 19:58:48.184008 28125 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:48.186090 28237 raft_consensus.cc:2802] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:48.186715 28237 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:48.188199 28201 ts_tablet_manager.cc:1428] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.159s user 0.002s sys 0.006s
I20250905 19:58:48.189363 28201 tablet_bootstrap.cc:492] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:48.192804 28226 consensus_queue.cc:1035] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:48.187258 28237 raft_consensus.cc:3058] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:48.176450 28200 tablet_bootstrap.cc:654] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
W20250905 19:58:48.225133 28235 log.cc:927] Time spent T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20250905 19:58:48.230862 28200 tablet_bootstrap.cc:492] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:48.234105 28200 ts_tablet_manager.cc:1397] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.120s user 0.013s sys 0.005s
I20250905 19:58:48.239384 28200 raft_consensus.cc:357] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.242511 28200 raft_consensus.cc:383] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.242903 28200 raft_consensus.cc:738] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.243646 28200 consensus_queue.cc:260] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.230933 28202 ts_tablet_manager.cc:1428] T 9f7f6e4e3a754dcda90928621168fbe7 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.114s user 0.024s sys 0.048s
I20250905 19:58:48.245371 28202 tablet_bootstrap.cc:492] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:48.250288 28237 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.251799 28202 tablet_bootstrap.cc:654] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:48.234267 28201 tablet_bootstrap.cc:654] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:48.254014 28200 ts_tablet_manager.cc:1428] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20250905 19:58:48.259621 28200 tablet_bootstrap.cc:492] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:48.264149 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:48.264889 28009 raft_consensus.cc:3058] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:48.274590 28009 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 19:58:48.279613 28125 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:58:48.279748 28237 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:48.280969 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:58:48.281714 28200 tablet_bootstrap.cc:654] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:48.280773 28237 raft_consensus.cc:2802] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election won for term 1
W20250905 19:58:48.282817 28122 leader_election.cc:343] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:48.282824 28237 raft_consensus.cc:695] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:58:48.285077 28237 consensus_queue.cc:237] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.288295 28201 tablet_bootstrap.cc:492] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:48.288587 28202 tablet_bootstrap.cc:492] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:48.289088 28201 ts_tablet_manager.cc:1397] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.100s user 0.010s sys 0.007s
I20250905 19:58:48.289697 28202 ts_tablet_manager.cc:1397] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.044s user 0.014s sys 0.018s
I20250905 19:58:48.292896 28201 raft_consensus.cc:357] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.293932 28201 raft_consensus.cc:383] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.294668 28201 raft_consensus.cc:738] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.293790 28202 raft_consensus.cc:357] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 19:58:48.295738 28202 raft_consensus.cc:383] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.295598 28201 consensus_queue.cc:260] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.296325 28202 raft_consensus.cc:738] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.297628 28202 consensus_queue.cc:260] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 19:58:48.298843 28201 ts_tablet_manager.cc:1428] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.009s user 0.006s sys 0.003s
I20250905 19:58:48.299979 28201 tablet_bootstrap.cc:492] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:48.302363 28202 ts_tablet_manager.cc:1428] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.012s user 0.009s sys 0.000s
I20250905 19:58:48.304121 28202 tablet_bootstrap.cc:492] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:48.311769 28201 tablet_bootstrap.cc:654] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:48.311919 28202 tablet_bootstrap.cc:654] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:48.318331 28238 raft_consensus.cc:491] T 9f7f6e4e3a754dcda90928621168fbe7 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:48.319155 28238 raft_consensus.cc:513] T 9f7f6e4e3a754dcda90928621168fbe7 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.323005 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:48.323311 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:58:48.324527 28051 leader_election.cc:343] T 9f7f6e4e3a754dcda90928621168fbe7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:48.324703 28238 leader_election.cc:290] T 9f7f6e4e3a754dcda90928621168fbe7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:48.327193 28201 tablet_bootstrap.cc:492] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:48.327713 28201 ts_tablet_manager.cc:1397] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.008s
I20250905 19:58:48.330798 28201 raft_consensus.cc:357] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:58:48.331235 28050 leader_election.cc:343] T 9f7f6e4e3a754dcda90928621168fbe7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:48.331598 28201 raft_consensus.cc:383] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.324527 28202 tablet_bootstrap.cc:492] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:48.332263 28202 ts_tablet_manager.cc:1397] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.028s user 0.014s sys 0.006s
I20250905 19:58:48.331733 28050 leader_election.cc:304] T 9f7f6e4e3a754dcda90928621168fbe7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:48.335335 28202 raft_consensus.cc:357] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.335577 28238 raft_consensus.cc:2747] T 9f7f6e4e3a754dcda90928621168fbe7 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:48.336139 28202 raft_consensus.cc:383] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.336544 28202 raft_consensus.cc:738] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.331913 28201 raft_consensus.cc:738] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.337409 28202 consensus_queue.cc:260] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.337505 27908 catalog_manager.cc:5582] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20250905 19:58:48.337906 28201 consensus_queue.cc:260] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.339872 28202 ts_tablet_manager.cc:1428] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20250905 19:58:48.295986 28200 tablet_bootstrap.cc:492] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:48.340484 28200 ts_tablet_manager.cc:1397] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.081s user 0.009s sys 0.005s
I20250905 19:58:48.341162 28237 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:48.341686 28237 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.343060 28202 tablet_bootstrap.cc:492] T 8afd77a8d80843aa805f971a2806ab86 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:48.344678 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:58:48.346309 28125 leader_election.cc:343] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:48.348510 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:48.350039 28202 tablet_bootstrap.cc:654] T 8afd77a8d80843aa805f971a2806ab86 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
W20250905 19:58:48.349973 28122 leader_election.cc:343] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:48.350613 28122 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:48.351542 28230 raft_consensus.cc:2747] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:48.352376 28237 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:48.352814 28201 ts_tablet_manager.cc:1428] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.025s user 0.003s sys 0.004s
I20250905 19:58:48.356137 28202 tablet_bootstrap.cc:492] T 8afd77a8d80843aa805f971a2806ab86 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:48.356676 28202 ts_tablet_manager.cc:1397] T 8afd77a8d80843aa805f971a2806ab86 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20250905 19:58:48.356482 28200 raft_consensus.cc:357] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.357288 28200 raft_consensus.cc:383] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.357578 28200 raft_consensus.cc:738] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.358196 28200 consensus_queue.cc:260] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.359437 28201 tablet_bootstrap.cc:492] T f2de38aacf1b4dccaeee999400fb6d75 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:48.359676 28202 raft_consensus.cc:357] T 8afd77a8d80843aa805f971a2806ab86 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.360376 28202 raft_consensus.cc:383] T 8afd77a8d80843aa805f971a2806ab86 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.360662 28202 raft_consensus.cc:738] T 8afd77a8d80843aa805f971a2806ab86 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.361519 28200 ts_tablet_manager.cc:1428] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.021s user 0.006s sys 0.001s
I20250905 19:58:48.361356 28202 consensus_queue.cc:260] T 8afd77a8d80843aa805f971a2806ab86 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.364351 28202 ts_tablet_manager.cc:1428] T 8afd77a8d80843aa805f971a2806ab86 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20250905 19:58:48.365543 28202 tablet_bootstrap.cc:492] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:48.366366 28200 tablet_bootstrap.cc:492] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:48.369189 28201 tablet_bootstrap.cc:654] T f2de38aacf1b4dccaeee999400fb6d75 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:48.372233 28202 tablet_bootstrap.cc:654] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:48.377732 28200 tablet_bootstrap.cc:654] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:48.386708 28200 tablet_bootstrap.cc:492] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:48.386988 28202 tablet_bootstrap.cc:492] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:48.387158 28200 ts_tablet_manager.cc:1397] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.002s
I20250905 19:58:48.387059 28201 tablet_bootstrap.cc:492] T f2de38aacf1b4dccaeee999400fb6d75 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:48.387611 28202 ts_tablet_manager.cc:1397] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.007s
I20250905 19:58:48.387785 28201 ts_tablet_manager.cc:1397] T f2de38aacf1b4dccaeee999400fb6d75 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.014s
I20250905 19:58:48.389801 28200 raft_consensus.cc:357] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.390563 28200 raft_consensus.cc:383] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.390913 28200 raft_consensus.cc:738] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.390607 28202 raft_consensus.cc:357] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.390846 28201 raft_consensus.cc:357] T f2de38aacf1b4dccaeee999400fb6d75 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.391466 28202 raft_consensus.cc:383] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.391649 28201 raft_consensus.cc:383] T f2de38aacf1b4dccaeee999400fb6d75 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.391836 28202 raft_consensus.cc:738] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.391986 28201 raft_consensus.cc:738] T f2de38aacf1b4dccaeee999400fb6d75 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.391764 28200 consensus_queue.cc:260] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.392660 28202 consensus_queue.cc:260] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.392767 28201 consensus_queue.cc:260] T f2de38aacf1b4dccaeee999400fb6d75 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.394533 28200 ts_tablet_manager.cc:1428] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250905 19:58:48.395031 28202 ts_tablet_manager.cc:1428] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250905 19:58:48.395678 28200 tablet_bootstrap.cc:492] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:48.397444 28202 tablet_bootstrap.cc:492] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:48.398703 28241 raft_consensus.cc:491] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:48.399190 28241 raft_consensus.cc:513] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.401407 28241 leader_election.cc:290] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:48.402256 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:48.402575 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:48.402848 28200 tablet_bootstrap.cc:654] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:48.402987 28084 raft_consensus.cc:2466] T 9f7f6e4e3a754dcda90928621168fbe7 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 0.
I20250905 19:58:48.403288 28159 raft_consensus.cc:2466] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 0.
I20250905 19:58:48.403503 28201 ts_tablet_manager.cc:1428] T f2de38aacf1b4dccaeee999400fb6d75 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20250905 19:58:48.404354 27973 leader_election.cc:304] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:48.405683 28241 raft_consensus.cc:2802] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:48.406080 28241 raft_consensus.cc:491] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:48.406467 28241 raft_consensus.cc:3058] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:48.410634 28202 tablet_bootstrap.cc:654] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:48.422322 28200 tablet_bootstrap.cc:492] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:48.422881 28200 ts_tablet_manager.cc:1397] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.027s user 0.019s sys 0.008s
I20250905 19:58:48.423626 28207 raft_consensus.cc:491] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:48.424315 28207 raft_consensus.cc:513] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.425557 28200 raft_consensus.cc:357] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.426337 28200 raft_consensus.cc:383] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.426767 28200 raft_consensus.cc:738] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.427424 28207 leader_election.cc:290] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:48.427487 28200 consensus_queue.cc:260] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.430284 28200 ts_tablet_manager.cc:1428] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250905 19:58:48.430483 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:48.431315 28200 tablet_bootstrap.cc:492] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:48.431422 28084 raft_consensus.cc:2466] T 8afd77a8d80843aa805f971a2806ab86 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 0.
I20250905 19:58:48.431703 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:48.434376 27973 leader_election.cc:304] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:48.435392 28207 raft_consensus.cc:2802] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:48.435818 28207 raft_consensus.cc:491] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:48.436582 28207 raft_consensus.cc:3058] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:48.437911 28200 tablet_bootstrap.cc:654] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
W20250905 19:58:48.438086 27975 leader_election.cc:343] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:48.404587 28201 tablet_bootstrap.cc:492] T 6dbd54765afc49cb862b267d6663000c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:48.450527 28200 tablet_bootstrap.cc:492] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:48.450059 28241 raft_consensus.cc:513] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.451011 28200 ts_tablet_manager.cc:1397] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.007s
I20250905 19:58:48.453687 28200 raft_consensus.cc:357] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.454063 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:58:48.454576 28200 raft_consensus.cc:383] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.454903 28200 raft_consensus.cc:738] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.455001 28084 raft_consensus.cc:3058] T 9f7f6e4e3a754dcda90928621168fbe7 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:48.455673 28200 consensus_queue.cc:260] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.456424 28207 raft_consensus.cc:513] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.459650 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:48.460246 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 19:58:48.461076 27975 leader_election.cc:343] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:48.461078 28085 raft_consensus.cc:3058] T 8afd77a8d80843aa805f971a2806ab86 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:48.461826 28200 ts_tablet_manager.cc:1428] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250905 19:58:48.462827 28200 tablet_bootstrap.cc:492] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:48.468623 28201 tablet_bootstrap.cc:654] T 6dbd54765afc49cb862b267d6663000c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:48.472885 28200 tablet_bootstrap.cc:654] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:48.474417 28241 leader_election.cc:290] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:48.475663 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:48.476416 28159 raft_consensus.cc:3058] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:48.476613 28085 raft_consensus.cc:2466] T 8afd77a8d80843aa805f971a2806ab86 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:58:48.477684 28207 leader_election.cc:290] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:48.478148 27973 leader_election.cc:304] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:58:48.479400 28207 raft_consensus.cc:2802] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:48.479993 28202 tablet_bootstrap.cc:492] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:48.480549 28202 ts_tablet_manager.cc:1397] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.083s user 0.029s sys 0.035s
I20250905 19:58:48.481236 28207 raft_consensus.cc:695] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:58:48.480041 28084 raft_consensus.cc:2466] T 9f7f6e4e3a754dcda90928621168fbe7 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:58:48.484596 28200 tablet_bootstrap.cc:492] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:48.485088 28200 ts_tablet_manager.cc:1397] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.008s
I20250905 19:58:48.485121 27973 leader_election.cc:304] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:48.485663 28159 raft_consensus.cc:2466] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:58:48.482378 28207 consensus_queue.cc:237] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.486361 28241 raft_consensus.cc:2802] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:48.486903 28241 raft_consensus.cc:695] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:58:48.487727 28241 consensus_queue.cc:237] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.488013 28200 raft_consensus.cc:357] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.488735 28200 raft_consensus.cc:383] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.488963 28200 raft_consensus.cc:738] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.489570 28200 consensus_queue.cc:260] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.494917 28200 ts_tablet_manager.cc:1428] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20250905 19:58:48.495910 28200 tablet_bootstrap.cc:492] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:48.483251 28202 raft_consensus.cc:357] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.497772 28202 raft_consensus.cc:383] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.498203 28202 raft_consensus.cc:738] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.499135 28202 consensus_queue.cc:260] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.501623 28201 tablet_bootstrap.cc:492] T 6dbd54765afc49cb862b267d6663000c P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:48.502137 28201 ts_tablet_manager.cc:1397] T 6dbd54765afc49cb862b267d6663000c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.098s user 0.005s sys 0.014s
I20250905 19:58:48.504691 28201 raft_consensus.cc:357] T 6dbd54765afc49cb862b267d6663000c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.505445 28201 raft_consensus.cc:383] T 6dbd54765afc49cb862b267d6663000c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.505699 28201 raft_consensus.cc:738] T 6dbd54765afc49cb862b267d6663000c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.506340 28201 consensus_queue.cc:260] T 6dbd54765afc49cb862b267d6663000c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.508641 28201 ts_tablet_manager.cc:1428] T 6dbd54765afc49cb862b267d6663000c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250905 19:58:48.508810 28202 ts_tablet_manager.cc:1428] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.028s user 0.008s sys 0.000s
I20250905 19:58:48.509600 28201 tablet_bootstrap.cc:492] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:48.510102 28202 tablet_bootstrap.cc:492] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:48.511905 27908 catalog_manager.cc:5582] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:48.513466 28200 tablet_bootstrap.cc:654] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:48.530347 28237 consensus_queue.cc:1035] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250905 19:58:48.545646 28122 consensus_peers.cc:489] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 -> Peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Couldn't send request to peer e4413ae5e0b64b52a6f360ac580239b9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:58:48.546077 28200 tablet_bootstrap.cc:492] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:48.518945 28202 tablet_bootstrap.cc:654] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:48.519043 28201 tablet_bootstrap.cc:654] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:48.549075 28238 raft_consensus.cc:491] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:48.549737 28238 raft_consensus.cc:513] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.551987 28238 leader_election.cc:290] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:48.549721 28200 ts_tablet_manager.cc:1397] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.054s user 0.014s sys 0.007s
I20250905 19:58:48.552794 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:48.553494 28009 raft_consensus.cc:2466] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:48.555362 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:48.555464 28200 raft_consensus.cc:357] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.556159 28200 raft_consensus.cc:383] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.556124 28159 raft_consensus.cc:2466] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:48.556515 28200 raft_consensus.cc:738] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.557201 28201 tablet_bootstrap.cc:492] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:48.557821 28201 ts_tablet_manager.cc:1397] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.048s user 0.010s sys 0.000s
I20250905 19:58:48.557763 28051 leader_election.cc:304] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:48.558620 28238 raft_consensus.cc:2802] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:48.558982 28238 raft_consensus.cc:491] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:48.557559 28200 consensus_queue.cc:260] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.559311 28238 raft_consensus.cc:3058] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:48.560549 28200 ts_tablet_manager.cc:1428] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250905 19:58:48.560489 28201 raft_consensus.cc:357] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 19:58:48.561287 28201 raft_consensus.cc:383] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.561579 28201 raft_consensus.cc:738] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.562371 28201 consensus_queue.cc:260] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 19:58:48.564829 28201 ts_tablet_manager.cc:1428] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20250905 19:58:48.566103 28201 tablet_bootstrap.cc:492] T 0adebfdecbae47298f234b849fcea7fc P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:48.567883 28200 tablet_bootstrap.cc:492] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:48.567514 28238 raft_consensus.cc:513] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.571805 28238 leader_election.cc:290] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:48.572705 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:48.572921 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:48.573360 28009 raft_consensus.cc:3058] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:48.573519 28159 raft_consensus.cc:3058] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:48.574226 28201 tablet_bootstrap.cc:654] T 0adebfdecbae47298f234b849fcea7fc P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:48.581426 28009 raft_consensus.cc:2466] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:48.589702 28051 leader_election.cc:304] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:48.590282 28202 tablet_bootstrap.cc:492] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:48.590783 28238 raft_consensus.cc:2802] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:48.591320 28238 raft_consensus.cc:695] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:58:48.586544 28159 raft_consensus.cc:2466] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:48.591645 27908 catalog_manager.cc:5582] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:48.592171 28238 consensus_queue.cc:237] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.591022 28202 ts_tablet_manager.cc:1397] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.081s user 0.017s sys 0.007s
I20250905 19:58:48.598116 28202 raft_consensus.cc:357] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.599386 28202 raft_consensus.cc:383] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.599752 28202 raft_consensus.cc:738] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.603440 28200 tablet_bootstrap.cc:654] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:48.605768 28201 tablet_bootstrap.cc:492] T 0adebfdecbae47298f234b849fcea7fc P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:48.600747 28202 consensus_queue.cc:260] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.606359 28201 ts_tablet_manager.cc:1397] T 0adebfdecbae47298f234b849fcea7fc P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.041s user 0.018s sys 0.017s
I20250905 19:58:48.608119 28202 ts_tablet_manager.cc:1428] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20250905 19:58:48.609149 28202 tablet_bootstrap.cc:492] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:48.609117 28201 raft_consensus.cc:357] T 0adebfdecbae47298f234b849fcea7fc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.609892 28201 raft_consensus.cc:383] T 0adebfdecbae47298f234b849fcea7fc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.610172 28201 raft_consensus.cc:738] T 0adebfdecbae47298f234b849fcea7fc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.610847 28201 consensus_queue.cc:260] T 0adebfdecbae47298f234b849fcea7fc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.613440 28201 ts_tablet_manager.cc:1428] T 0adebfdecbae47298f234b849fcea7fc P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250905 19:58:48.614670 28201 tablet_bootstrap.cc:492] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:48.617832 28209 raft_consensus.cc:491] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:48.618413 28209 raft_consensus.cc:513] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.620651 28209 leader_election.cc:290] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:48.621656 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:48.624457 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:48.626296 28050 leader_election.cc:304] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:48.628382 28209 raft_consensus.cc:2747] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:48.628818 28202 tablet_bootstrap.cc:654] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:48.629567 28201 tablet_bootstrap.cc:654] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:48.638947 28202 tablet_bootstrap.cc:492] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:48.639640 28202 ts_tablet_manager.cc:1397] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.000s
I20250905 19:58:48.642546 28202 raft_consensus.cc:357] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.643286 28202 raft_consensus.cc:383] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.643569 28202 raft_consensus.cc:738] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.643473 28230 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:48.643957 28201 tablet_bootstrap.cc:492] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:48.644536 28201 ts_tablet_manager.cc:1397] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.007s
I20250905 19:58:48.644299 28202 consensus_queue.cc:260] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.644244 28230 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.648641 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:48.648253 28201 raft_consensus.cc:357] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.649394 28009 raft_consensus.cc:2466] T 0adebfdecbae47298f234b849fcea7fc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 0.
I20250905 19:58:48.650130 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:48.649377 28201 raft_consensus.cc:383] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.651341 28201 raft_consensus.cc:738] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.651751 28125 leader_election.cc:304] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:58:48.653039 28206 raft_consensus.cc:2802] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:48.653565 28206 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:48.654018 28206 raft_consensus.cc:3058] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:48.655699 28202 ts_tablet_manager.cc:1428] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20250905 19:58:48.655905 28201 consensus_queue.cc:260] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.656937 28230 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 19:58:48.656404 28122 leader_election.cc:343] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:48.658182 28202 tablet_bootstrap.cc:492] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:48.658948 28201 ts_tablet_manager.cc:1428] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.014s user 0.007s sys 0.004s
I20250905 19:58:48.659937 28201 tablet_bootstrap.cc:492] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:48.666647 28230 raft_consensus.cc:491] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:48.667383 28201 tablet_bootstrap.cc:654] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:48.668180 28202 tablet_bootstrap.cc:654] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:48.667680 28230 raft_consensus.cc:513] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.671990 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:58:48.674032 28125 leader_election.cc:343] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:58:48.676589 27956 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250905 19:58:48.679597 28201 tablet_bootstrap.cc:492] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:48.680719 28201 ts_tablet_manager.cc:1397] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.008s
I20250905 19:58:48.682859 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:48.683856 28084 raft_consensus.cc:2466] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 0.
I20250905 19:58:48.685564 28122 leader_election.cc:304] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:48.686961 28237 raft_consensus.cc:2802] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:48.687479 28237 raft_consensus.cc:491] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:48.687949 28237 raft_consensus.cc:3058] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:48.687767 28201 raft_consensus.cc:357] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.688563 28201 raft_consensus.cc:383] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.688855 28201 raft_consensus.cc:738] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.690062 28201 consensus_queue.cc:260] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.691285 28209 raft_consensus.cc:491] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:48.691994 28209 raft_consensus.cc:513] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.693300 28201 ts_tablet_manager.cc:1428] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20250905 19:58:48.693300 28202 tablet_bootstrap.cc:492] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:48.694662 28201 tablet_bootstrap.cc:492] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:48.694825 28202 ts_tablet_manager.cc:1397] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.038s user 0.014s sys 0.002s
I20250905 19:58:48.696117 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:48.696789 28009 raft_consensus.cc:2466] T f2de38aacf1b4dccaeee999400fb6d75 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:48.696888 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:48.697381 28202 raft_consensus.cc:357] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.698287 28202 raft_consensus.cc:383] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.698743 28202 raft_consensus.cc:738] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.699695 28202 consensus_queue.cc:260] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:58:48.700356 28050 leader_election.cc:343] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:48.701226 28209 leader_election.cc:290] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:48.702127 28201 tablet_bootstrap.cc:654] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:48.702756 28051 leader_election.cc:304] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:58:48.709482 28276 raft_consensus.cc:2802] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:48.710131 28276 raft_consensus.cc:491] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:48.715340 28276 raft_consensus.cc:3058] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:48.712604 28202 ts_tablet_manager.cc:1428] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20250905 19:58:48.714692 28230 leader_election.cc:290] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:48.721210 28202 tablet_bootstrap.cc:492] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:48.726080 28238 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:48.727227 27908 catalog_manager.cc:5582] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:48.713570 28237 raft_consensus.cc:513] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.731678 28237 leader_election.cc:290] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:48.731585 28206 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.734014 28201 tablet_bootstrap.cc:492] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:48.734556 28201 ts_tablet_manager.cc:1397] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.040s user 0.019s sys 0.008s
I20250905 19:58:48.736231 28206 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:48.736650 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:58:48.737001 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:58:48.737521 28084 raft_consensus.cc:3058] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:58:48.738217 28122 leader_election.cc:343] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:48.738931 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:48.739573 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:48.739984 28010 raft_consensus.cc:3058] T 0adebfdecbae47298f234b849fcea7fc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:48.728307 28238 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.744168 28209 raft_consensus.cc:491] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:48.744671 28209 raft_consensus.cc:513] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 19:58:48.745494 28238 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:48.748608 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:48.752599 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:48.753324 28159 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:48.753505 28010 raft_consensus.cc:2466] T 0adebfdecbae47298f234b849fcea7fc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 19:58:48.753893 28084 raft_consensus.cc:2466] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 19:58:48.754832 28125 leader_election.cc:304] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:48.755502 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:48.756126 28050 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:48.750212 28209 leader_election.cc:290] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421)
I20250905 19:58:48.756800 28202 tablet_bootstrap.cc:654] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:48.756150 28159 raft_consensus.cc:2466] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
W20250905 19:58:48.757113 28051 leader_election.cc:343] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:48.758647 28050 leader_election.cc:304] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:48.759364 28200 tablet_bootstrap.cc:492] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:48.759845 28200 ts_tablet_manager.cc:1397] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.192s user 0.045s sys 0.086s
I20250905 19:58:48.759871 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:48.760743 28008 raft_consensus.cc:2466] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:48.761116 28122 leader_election.cc:304] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:48.758080 28206 raft_consensus.cc:2802] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:48.762190 28237 raft_consensus.cc:2802] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:48.762820 28237 raft_consensus.cc:695] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:58:48.763193 28206 raft_consensus.cc:695] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
W20250905 19:58:48.763574 28125 leader_election.cc:343] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:48.764006 28200 raft_consensus.cc:357] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.764549 28209 raft_consensus.cc:2802] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:48.764206 28206 consensus_queue.cc:237] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.764787 28200 raft_consensus.cc:383] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.765013 28209 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:48.765316 28200 raft_consensus.cc:738] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.765432 28209 raft_consensus.cc:3058] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:48.766358 28200 consensus_queue.cc:260] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.767426 28237 consensus_queue.cc:237] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.769393 28200 ts_tablet_manager.cc:1428] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20250905 19:58:48.771092 28238 raft_consensus.cc:2802] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:48.775310 28201 raft_consensus.cc:357] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.776002 28200 tablet_bootstrap.cc:492] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:48.779949 28276 raft_consensus.cc:513] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.782060 28201 raft_consensus.cc:383] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.782881 28201 raft_consensus.cc:738] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.783756 28201 consensus_queue.cc:260] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.784508 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:48.784777 28276 leader_election.cc:290] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:48.785115 28009 raft_consensus.cc:3058] T f2de38aacf1b4dccaeee999400fb6d75 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:48.786134 28201 ts_tablet_manager.cc:1428] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.051s user 0.006s sys 0.000s
I20250905 19:58:48.786104 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:48.787290 28201 tablet_bootstrap.cc:492] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:48.776710 28238 raft_consensus.cc:491] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250905 19:58:48.789784 28050 leader_election.cc:343] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:48.791648 28238 raft_consensus.cc:3058] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:48.801196 28238 raft_consensus.cc:513] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 19:58:48.805449 28201 tablet_bootstrap.cc:654] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:48.806388 28202 tablet_bootstrap.cc:492] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:48.798956 28209 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.807046 28202 ts_tablet_manager.cc:1397] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.086s user 0.014s sys 0.008s
I20250905 19:58:48.809875 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:48.807503 27906 catalog_manager.cc:5582] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
W20250905 19:58:48.811443 28051 leader_election.cc:343] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:48.806950 28009 raft_consensus.cc:2466] T f2de38aacf1b4dccaeee999400fb6d75 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:48.813215 28051 leader_election.cc:304] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:58:48.814085 28276 raft_consensus.cc:2802] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:48.816135 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:48.816778 28159 raft_consensus.cc:3058] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:48.810853 28202 raft_consensus.cc:357] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.817497 28200 tablet_bootstrap.cc:654] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:48.817656 28202 raft_consensus.cc:383] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.818037 28202 raft_consensus.cc:738] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.818945 28202 consensus_queue.cc:260] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.821774 28276 raft_consensus.cc:695] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:58:48.822829 28276 consensus_queue.cc:237] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.820627 28209 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:48.832208 28237 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:48.835181 28237 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.838590 28237 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:48.840090 28282 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:48.840410 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:48.840669 28282 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.841934 28084 raft_consensus.cc:2466] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 0.
I20250905 19:58:48.842353 28241 raft_consensus.cc:491] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:48.843236 28202 ts_tablet_manager.cc:1428] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.036s user 0.007s sys 0.008s
I20250905 19:58:48.842893 28241 raft_consensus.cc:513] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.844934 28241 leader_election.cc:290] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:48.844770 28122 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:48.845985 28237 raft_consensus.cc:2802] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:48.845907 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:48.846591 28237 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:48.847007 28237 raft_consensus.cc:3058] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:48.847041 28009 raft_consensus.cc:2466] T 6dbd54765afc49cb862b267d6663000c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 0.
I20250905 19:58:48.850782 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:48.852089 28157 raft_consensus.cc:2466] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 0.
I20250905 19:58:48.855947 28202 tablet_bootstrap.cc:492] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:48.854099 27975 leader_election.cc:304] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:58:48.858111 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:48.858770 28157 raft_consensus.cc:3058] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:48.859189 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:48.859828 28084 raft_consensus.cc:2466] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 0.
I20250905 19:58:48.860008 28238 leader_election.cc:290] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421)
I20250905 19:58:48.860265 28201 tablet_bootstrap.cc:492] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:48.860826 28241 raft_consensus.cc:2802] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:48.861377 28241 raft_consensus.cc:491] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:48.861769 28241 raft_consensus.cc:3058] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:48.860906 28201 ts_tablet_manager.cc:1397] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.074s user 0.011s sys 0.015s
I20250905 19:58:48.854169 28159 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:48.870604 28050 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:48.869632 28201 raft_consensus.cc:357] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.872144 28201 raft_consensus.cc:383] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.872306 28238 raft_consensus.cc:2802] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:48.872485 28201 raft_consensus.cc:738] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.873284 28201 consensus_queue.cc:260] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.872888 28238 raft_consensus.cc:695] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:58:48.878867 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:48.879547 28009 raft_consensus.cc:3058] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:48.879866 28238 consensus_queue.cc:237] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.880666 27906 catalog_manager.cc:5582] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:48.883194 28157 raft_consensus.cc:2466] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:48.883749 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:48.884429 28008 raft_consensus.cc:2466] T 6dbd54765afc49cb862b267d6663000c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:48.884940 28202 tablet_bootstrap.cc:654] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:48.891086 28201 ts_tablet_manager.cc:1428] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.028s user 0.006s sys 0.003s
I20250905 19:58:48.872840 28282 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:48.893150 28201 tablet_bootstrap.cc:492] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:48.890229 28051 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:48.897089 28226 consensus_queue.cc:1035] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:48.898191 28282 raft_consensus.cc:2802] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:48.898612 28282 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:48.898980 28282 raft_consensus.cc:3058] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:48.890146 28050 leader_election.cc:304] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:48.902765 28009 raft_consensus.cc:2466] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:48.891347 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:58:48.905719 28122 consensus_peers.cc:489] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 -> Peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Couldn't send request to peer e4413ae5e0b64b52a6f360ac580239b9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:58:48.916566 28201 tablet_bootstrap.cc:654] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:48.869619 28237 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.909968 28238 raft_consensus.cc:2802] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:48.923269 28238 raft_consensus.cc:695] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:58:48.924815 28238 consensus_queue.cc:237] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 19:58:48.926079 28206 consensus_queue.cc:1035] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:48.949553 28200 tablet_bootstrap.cc:492] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:48.950572 28200 ts_tablet_manager.cc:1397] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.175s user 0.015s sys 0.014s
I20250905 19:58:48.954288 28159 raft_consensus.cc:2391] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 1: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:58:48.970597 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:48.971330 28009 raft_consensus.cc:3058] T 6dbd54765afc49cb862b267d6663000c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:48.971110 28241 raft_consensus.cc:513] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.975122 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:58:48.975194 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:48.975060 28200 raft_consensus.cc:357] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.976475 28200 raft_consensus.cc:383] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:48.976665 28200 raft_consensus.cc:738] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:48.977133 28200 consensus_queue.cc:260] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.978634 28241 leader_election.cc:290] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:48.978652 28159 raft_consensus.cc:3058] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:48.980589 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:58:48.981523 28085 raft_consensus.cc:3058] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:48.982122 28200 ts_tablet_manager.cc:1428] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.031s user 0.003s sys 0.002s
I20250905 19:58:48.982782 28230 raft_consensus.cc:491] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:48.983186 28200 tablet_bootstrap.cc:492] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:48.983263 28230 raft_consensus.cc:513] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.986186 28230 leader_election.cc:290] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:48.978169 28237 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:48.988864 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:48.989694 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:48.990607 28083 raft_consensus.cc:2391] T 8afd77a8d80843aa805f971a2806ab86 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 1: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:58:48.991581 28122 leader_election.cc:304] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:48.991647 28238 consensus_queue.cc:1035] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:48.992053 28202 tablet_bootstrap.cc:492] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:48.992543 28202 ts_tablet_manager.cc:1397] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.137s user 0.023s sys 0.026s
I20250905 19:58:48.992795 28276 raft_consensus.cc:491] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:48.992871 28230 raft_consensus.cc:3058] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:48.993794 28276 raft_consensus.cc:513] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:48.996552 28276 leader_election.cc:290] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:48.998301 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:48.999400 28008 raft_consensus.cc:2466] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:49.000968 28051 leader_election.cc:304] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:48.998623 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:49.001884 28085 raft_consensus.cc:2466] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:58:49.003096 28202 raft_consensus.cc:357] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.003602 28202 raft_consensus.cc:383] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:49.003808 28202 raft_consensus.cc:738] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:49.004938 28202 consensus_queue.cc:260] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.001924 28230 raft_consensus.cc:2747] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:49.002628 27907 catalog_manager.cc:5582] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:49.006971 28276 raft_consensus.cc:2802] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:49.010385 28276 raft_consensus.cc:491] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:48.984256 28009 raft_consensus.cc:2466] T 6dbd54765afc49cb862b267d6663000c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 19:58:49.007267 28159 raft_consensus.cc:2466] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:58:49.015605 28201 tablet_bootstrap.cc:492] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:49.016085 28201 ts_tablet_manager.cc:1397] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.123s user 0.017s sys 0.005s
I20250905 19:58:49.016373 28226 raft_consensus.cc:491] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:49.016996 28226 raft_consensus.cc:513] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.018767 28201 raft_consensus.cc:357] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.019452 28201 raft_consensus.cc:383] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:49.019763 28201 raft_consensus.cc:738] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:49.020018 28226 leader_election.cc:290] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:48.982750 27906 catalog_manager.cc:5582] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20250905 19:58:49.020449 28201 consensus_queue.cc:260] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:58:49.007541 28050 leader_election.cc:343] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:49.010684 28241 raft_consensus.cc:491] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:49.027092 27973 leader_election.cc:304] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:49.027525 28241 raft_consensus.cc:513] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.010808 28276 raft_consensus.cc:3058] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:49.010690 27907 catalog_manager.cc:5582] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } }
I20250905 19:58:49.024173 28201 ts_tablet_manager.cc:1428] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20250905 19:58:49.030153 28201 tablet_bootstrap.cc:492] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:49.019541 28282 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.032162 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:49.033066 28202 ts_tablet_manager.cc:1428] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.040s user 0.004s sys 0.003s
I20250905 19:58:49.036712 28202 tablet_bootstrap.cc:492] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:49.039636 28241 leader_election.cc:290] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:49.041528 28241 raft_consensus.cc:2802] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:49.043381 28084 raft_consensus.cc:2391] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 1: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:58:49.045939 28241 raft_consensus.cc:695] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:58:49.047366 28125 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:58:49.048601 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:49.049300 28230 raft_consensus.cc:2802] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:49.049484 28084 raft_consensus.cc:2391] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 1: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:58:49.050051 28230 raft_consensus.cc:695] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
W20250905 19:58:49.050699 27973 leader_election.cc:343] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:49.051698 28241 consensus_queue.cc:237] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.051031 28230 consensus_queue.cc:237] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.054327 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:49.054780 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:49.052593 28296 consensus_queue.cc:1035] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20250905 19:58:49.056191 28207 consensus_queue.cc:1035] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:49.060302 28282 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:49.058957 27975 leader_election.cc:304] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:49.061115 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:49.061611 28200 tablet_bootstrap.cc:654] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:49.063961 28294 consensus_queue.cc:1035] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250905 19:58:49.067322 28207 raft_consensus.cc:3058] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:49.101253 28296 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:49.101806 28296 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:58:49.102710 27975 leader_election.cc:343] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:49.106333 28209 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:49.107460 28209 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.106727 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:49.108981 28009 raft_consensus.cc:2391] T 6dbd54765afc49cb862b267d6663000c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 1: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:58:49.109274 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:49.103925 28296 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:49.103264 27975 leader_election.cc:304] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:49.118090 28276 raft_consensus.cc:513] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.123983 28209 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:49.125320 28282 consensus_queue.cc:1035] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:49.126979 28207 raft_consensus.cc:2747] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:49.127590 28202 tablet_bootstrap.cc:654] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:49.131608 28276 leader_election.cc:290] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:49.177636 28298 raft_consensus.cc:2747] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:49.138903 28201 tablet_bootstrap.cc:654] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:49.136922 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:49.232559 28237 consensus_queue.cc:1035] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.050s
I20250905 19:58:49.134456 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:49.137080 28083 raft_consensus.cc:3058] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:49.223178 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:58:49.266686 28050 consensus_peers.cc:489] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 -> Peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Couldn't send request to peer 26b2f874b8e54f08b01a9ed00d5cce05. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:58:49.287874 28155 raft_consensus.cc:2391] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 1: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
W20250905 19:58:49.288466 28050 leader_election.cc:343] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:49.177857 28051 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:49.297667 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:49.300884 28007 raft_consensus.cc:3058] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:49.304706 28202 tablet_bootstrap.cc:492] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:49.310326 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:49.306146 28276 raft_consensus.cc:2747] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250905 19:58:49.315069 27908 catalog_manager.cc:5582] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:49.323550 28202 ts_tablet_manager.cc:1397] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.273s user 0.014s sys 0.001s
W20250905 19:58:49.321722 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 144026us
I20250905 19:58:49.326495 28276 consensus_queue.cc:1035] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:49.302565 28237 consensus_queue.cc:1035] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20250905 19:58:49.345551 28007 raft_consensus.cc:2466] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:49.347512 28237 consensus_queue.cc:1035] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:49.352540 28202 raft_consensus.cc:357] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.353457 28202 raft_consensus.cc:383] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:49.353726 28202 raft_consensus.cc:738] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:49.354228 28202 consensus_queue.cc:260] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.350826 27975 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:49.358798 28051 leader_election.cc:304] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:49.360071 28298 raft_consensus.cc:3058] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:49.361784 28282 raft_consensus.cc:2747] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:49.367821 28202 ts_tablet_manager.cc:1428] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.043s user 0.006s sys 0.000s
I20250905 19:58:49.368501 28206 raft_consensus.cc:491] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:49.368878 28202 tablet_bootstrap.cc:492] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:49.369004 28206 raft_consensus.cc:513] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.387681 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:49.387961 28206 leader_election.cc:290] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:49.388655 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:58:49.389463 28125 leader_election.cc:343] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:49.390341 28200 tablet_bootstrap.cc:492] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:49.390889 28200 ts_tablet_manager.cc:1397] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.408s user 0.066s sys 0.057s
W20250905 19:58:49.390784 28122 leader_election.cc:343] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:49.391510 28082 raft_consensus.cc:3058] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:49.391561 28122 leader_election.cc:304] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:49.402626 28201 tablet_bootstrap.cc:492] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:49.403237 28201 ts_tablet_manager.cc:1397] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.373s user 0.030s sys 0.057s
I20250905 19:58:49.350379 27907 catalog_manager.cc:5582] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20250905 19:58:49.406414 28051 leader_election.cc:304] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:58:49.411792 28200 raft_consensus.cc:357] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.412863 28200 raft_consensus.cc:383] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:49.413640 28282 raft_consensus.cc:2802] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:49.413954 28200 raft_consensus.cc:738] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:49.414783 28200 consensus_queue.cc:260] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.414173 28282 raft_consensus.cc:695] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:58:49.404295 28237 raft_consensus.cc:2747] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:49.423338 28282 consensus_queue.cc:237] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.465381 28300 consensus_queue.cc:1035] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250905 19:58:49.480785 28201 raft_consensus.cc:357] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.481581 28201 raft_consensus.cc:383] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:49.481813 28201 raft_consensus.cc:738] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:49.484920 28200 ts_tablet_manager.cc:1428] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.094s user 0.005s sys 0.003s
I20250905 19:58:49.485752 28200 tablet_bootstrap.cc:492] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:49.483773 28201 consensus_queue.cc:260] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.491694 28312 consensus_queue.cc:1035] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20250905 19:58:49.482690 28230 consensus_queue.cc:1035] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20250905 19:58:49.495692 28201 ts_tablet_manager.cc:1428] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.092s user 0.006s sys 0.010s
I20250905 19:58:49.496224 28298 raft_consensus.cc:2747] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:49.496703 28201 tablet_bootstrap.cc:492] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:49.509824 28284 consensus_queue.cc:1035] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20250905 19:58:49.501474 28276 consensus_queue.cc:1035] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20250905 19:58:49.521895 28312 raft_consensus.cc:491] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:49.522579 28312 raft_consensus.cc:513] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.533633 28282 consensus_queue.cc:1035] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:49.528585 28276 raft_consensus.cc:491] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:49.536235 28276 raft_consensus.cc:513] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.545987 28282 consensus_queue.cc:1035] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:49.547401 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:49.551710 28237 consensus_queue.cc:1035] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
W20250905 19:58:49.555404 28308 log.cc:927] Time spent T 0adebfdecbae47298f234b849fcea7fc P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.064s user 0.005s sys 0.009s
W20250905 19:58:49.562494 28051 leader_election.cc:343] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:49.550776 28276 leader_election.cc:290] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:49.564627 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:49.577522 28202 tablet_bootstrap.cc:654] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
W20250905 19:58:49.578876 28050 leader_election.cc:343] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:49.579934 28312 leader_election.cc:290] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:49.585309 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:49.595832 28050 leader_election.cc:304] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:49.607112 28276 raft_consensus.cc:2747] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:49.611651 28294 consensus_queue.cc:1035] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250905 19:58:49.633937 28322 log.cc:927] Time spent T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
I20250905 19:58:49.634944 28201 tablet_bootstrap.cc:654] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:49.668589 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:49.670729 28125 leader_election.cc:304] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:49.680096 28209 raft_consensus.cc:491] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:49.680943 28209 raft_consensus.cc:513] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.684661 28209 leader_election.cc:290] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:49.685632 28312 raft_consensus.cc:2747] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:49.712747 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:49.713693 28006 raft_consensus.cc:2466] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:49.717648 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:49.720175 28051 leader_election.cc:304] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:49.721382 28209 raft_consensus.cc:2802] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250905 19:58:49.721777 28050 leader_election.cc:343] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:49.721906 28209 raft_consensus.cc:491] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:49.722597 28209 raft_consensus.cc:3058] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:58:49.733466 28321 log.cc:927] Time spent T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.093s user 0.001s sys 0.000s
I20250905 19:58:49.734179 28209 raft_consensus.cc:513] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.736598 28200 tablet_bootstrap.cc:654] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:49.798506 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:49.799226 28007 raft_consensus.cc:3058] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:58:49.808168 27956 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250905 19:58:49.809204 28155 raft_consensus.cc:3058] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:49.810246 28202 tablet_bootstrap.cc:492] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:49.810860 28202 ts_tablet_manager.cc:1397] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.442s user 0.037s sys 0.035s
I20250905 19:58:49.813074 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:49.813077 28294 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:49.814664 28209 leader_election.cc:290] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:58:49.814801 28050 leader_election.cc:343] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:49.814807 28294 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.816545 28201 tablet_bootstrap.cc:492] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:49.817040 28201 ts_tablet_manager.cc:1397] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.321s user 0.021s sys 0.008s
I20250905 19:58:49.820447 28201 raft_consensus.cc:357] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.821173 28201 raft_consensus.cc:383] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:49.821100 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:49.821575 28201 raft_consensus.cc:738] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:49.821050 28202 raft_consensus.cc:357] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.822316 28201 consensus_queue.cc:260] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.822671 28202 raft_consensus.cc:383] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:49.823190 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:49.823104 28202 raft_consensus.cc:738] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:49.824527 28201 ts_tablet_manager.cc:1428] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250905 19:58:49.824357 28202 consensus_queue.cc:260] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:58:49.824923 27975 leader_election.cc:343] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:49.825182 28007 raft_consensus.cc:2466] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:49.825241 28084 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 0.
I20250905 19:58:49.825557 28201 tablet_bootstrap.cc:492] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:49.825675 28294 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:49.828017 28200 tablet_bootstrap.cc:492] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:49.834007 28200 ts_tablet_manager.cc:1397] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.348s user 0.032s sys 0.010s
I20250905 19:58:49.837188 27973 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:58:49.832731 28051 leader_election.cc:304] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:58:49.838449 28295 raft_consensus.cc:2802] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:49.838631 28282 raft_consensus.cc:2802] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:49.839012 28295 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:49.839295 28282 raft_consensus.cc:695] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:58:49.839519 28295 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:49.840386 28282 consensus_queue.cc:237] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.837033 28200 raft_consensus.cc:357] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.843988 28200 raft_consensus.cc:383] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:49.844360 28200 raft_consensus.cc:738] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:49.845110 28201 tablet_bootstrap.cc:654] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:49.857090 28200 consensus_queue.cc:260] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.859686 28201 tablet_bootstrap.cc:492] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:49.860198 28201 ts_tablet_manager.cc:1397] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.009s
I20250905 19:58:49.860458 28200 ts_tablet_manager.cc:1428] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.026s user 0.010s sys 0.000s
I20250905 19:58:49.861838 28200 tablet_bootstrap.cc:492] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:49.862747 28201 raft_consensus.cc:357] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.863481 28201 raft_consensus.cc:383] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:49.863765 28201 raft_consensus.cc:738] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:49.864622 28201 consensus_queue.cc:260] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.867172 28201 ts_tablet_manager.cc:1428] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250905 19:58:49.868170 28201 tablet_bootstrap.cc:492] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:49.870622 28202 ts_tablet_manager.cc:1428] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.059s user 0.005s sys 0.008s
I20250905 19:58:49.872156 28202 tablet_bootstrap.cc:492] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:49.874169 28201 tablet_bootstrap.cc:654] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:49.877358 28200 tablet_bootstrap.cc:654] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:49.880151 28202 tablet_bootstrap.cc:654] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:49.884881 27907 catalog_manager.cc:5582] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:49.891181 28295 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.893680 28295 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:49.894744 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:58:49.896610 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:49.896795 28084 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:58:49.898370 27975 leader_election.cc:343] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:49.908248 28084 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:58:49.911923 27973 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:58:49.914026 28295 raft_consensus.cc:2802] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:49.914830 28295 raft_consensus.cc:695] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:58:49.915755 28295 consensus_queue.cc:237] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.919720 28206 raft_consensus.cc:491] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:49.920320 28206 raft_consensus.cc:513] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.925241 28206 leader_election.cc:290] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:49.927098 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:49.927966 28007 raft_consensus.cc:2391] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 1: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:58:49.928506 28201 tablet_bootstrap.cc:492] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:49.928982 28201 ts_tablet_manager.cc:1397] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.061s user 0.014s sys 0.003s
I20250905 19:58:49.932250 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:49.934115 28122 leader_election.cc:304] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:49.935015 28206 raft_consensus.cc:3058] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:49.939965 28201 raft_consensus.cc:357] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.940776 28201 raft_consensus.cc:383] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:49.941071 28201 raft_consensus.cc:738] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:49.941735 28201 consensus_queue.cc:260] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.945236 28294 raft_consensus.cc:491] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:49.945725 28294 raft_consensus.cc:513] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.947110 28201 ts_tablet_manager.cc:1428] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.018s user 0.006s sys 0.002s
I20250905 19:58:49.947938 28294 leader_election.cc:290] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:49.949597 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:49.949553 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:49.950744 28156 raft_consensus.cc:2466] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 0.
I20250905 19:58:49.952178 27975 leader_election.cc:304] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 19:58:49.953899 27973 leader_election.cc:343] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:49.955092 28294 raft_consensus.cc:2802] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:49.955536 28294 raft_consensus.cc:491] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:49.955971 28294 raft_consensus.cc:3058] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:49.956038 28206 raft_consensus.cc:2747] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:49.957971 28201 tablet_bootstrap.cc:492] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:49.963261 28241 raft_consensus.cc:491] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:49.963827 28241 raft_consensus.cc:513] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.970371 28200 tablet_bootstrap.cc:492] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:49.970942 28200 ts_tablet_manager.cc:1397] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.109s user 0.011s sys 0.026s
I20250905 19:58:49.973948 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:49.975971 28201 tablet_bootstrap.cc:654] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
W20250905 19:58:49.975914 27973 leader_election.cc:343] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:49.976833 28200 raft_consensus.cc:357] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.977162 28294 raft_consensus.cc:513] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.977552 28200 raft_consensus.cc:383] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:49.978016 28200 raft_consensus.cc:738] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:49.978807 28200 consensus_queue.cc:260] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:49.980425 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 19:58:49.982355 27973 leader_election.cc:343] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:49.987741 28200 ts_tablet_manager.cc:1428] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.016s user 0.005s sys 0.001s
I20250905 19:58:49.988814 28200 tablet_bootstrap.cc:492] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:49.988948 28282 consensus_queue.cc:1035] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:49.994545 28202 tablet_bootstrap.cc:492] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:49.995085 28202 ts_tablet_manager.cc:1397] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.123s user 0.013s sys 0.002s
I20250905 19:58:49.996580 28294 leader_election.cc:290] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:49.996851 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:49.997602 28156 raft_consensus.cc:3058] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:49.980475 27907 catalog_manager.cc:5582] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:50.010314 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:58:50.012135 27975 leader_election.cc:343] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:50.012650 27975 leader_election.cc:304] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:50.013491 28295 raft_consensus.cc:2747] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:50.013593 28241 leader_election.cc:290] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:50.021950 28202 raft_consensus.cc:357] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.022574 28202 raft_consensus.cc:383] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:50.022859 28202 raft_consensus.cc:738] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:50.023558 28156 raft_consensus.cc:2466] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:58:50.023526 28202 consensus_queue.cc:260] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.023355 27905 catalog_manager.cc:5582] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:50.025105 27975 leader_election.cc:304] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:50.026086 28202 ts_tablet_manager.cc:1428] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.031s user 0.004s sys 0.006s
I20250905 19:58:50.026520 28295 raft_consensus.cc:2802] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:50.027164 28202 tablet_bootstrap.cc:492] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:50.030508 28206 raft_consensus.cc:491] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:50.031055 28206 raft_consensus.cc:513] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.031819 28200 tablet_bootstrap.cc:654] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:50.032595 28295 raft_consensus.cc:695] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:58:50.033144 28206 leader_election.cc:290] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:50.035713 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:50.037010 28276 consensus_queue.cc:1035] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:50.037374 28202 tablet_bootstrap.cc:654] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:50.039451 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:50.040345 28006 raft_consensus.cc:2391] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 1: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:58:50.038902 28295 consensus_queue.cc:237] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.043226 28201 tablet_bootstrap.cc:492] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:50.043731 28201 ts_tablet_manager.cc:1397] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.086s user 0.023s sys 0.009s
I20250905 19:58:50.043011 28125 leader_election.cc:304] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:50.054232 28202 tablet_bootstrap.cc:492] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:50.055095 28202 ts_tablet_manager.cc:1397] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.028s user 0.014s sys 0.000s
I20250905 19:58:50.056493 28201 raft_consensus.cc:357] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.057886 28237 raft_consensus.cc:3058] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:50.058248 28201 raft_consensus.cc:383] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:50.058578 28201 raft_consensus.cc:738] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:50.058662 28202 raft_consensus.cc:357] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.059286 28201 consensus_queue.cc:260] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.059646 28202 raft_consensus.cc:383] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:50.061561 28201 ts_tablet_manager.cc:1428] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.018s user 0.004s sys 0.002s
I20250905 19:58:50.062628 28201 tablet_bootstrap.cc:492] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:50.060093 28202 raft_consensus.cc:738] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:50.063549 28202 consensus_queue.cc:260] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.067078 28202 ts_tablet_manager.cc:1428] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.012s user 0.009s sys 0.000s
I20250905 19:58:50.068498 28202 tablet_bootstrap.cc:492] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:50.080358 28237 raft_consensus.cc:2747] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:50.080581 28202 tablet_bootstrap.cc:654] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:50.088749 28201 tablet_bootstrap.cc:654] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:50.091373 28200 tablet_bootstrap.cc:492] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:50.091842 28200 ts_tablet_manager.cc:1397] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.103s user 0.017s sys 0.027s
I20250905 19:58:50.096276 28201 tablet_bootstrap.cc:492] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:50.096145 28200 raft_consensus.cc:357] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.096769 28201 ts_tablet_manager.cc:1397] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.002s
I20250905 19:58:50.097085 28200 raft_consensus.cc:383] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:50.097440 28200 raft_consensus.cc:738] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:50.098143 28200 consensus_queue.cc:260] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.100406 28200 ts_tablet_manager.cc:1428] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250905 19:58:50.101749 28200 tablet_bootstrap.cc:492] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:50.103403 28201 raft_consensus.cc:357] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.104117 28201 raft_consensus.cc:383] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:50.104405 28201 raft_consensus.cc:738] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:50.105180 28201 consensus_queue.cc:260] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.107497 28201 ts_tablet_manager.cc:1428] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20250905 19:58:50.108382 28201 tablet_bootstrap.cc:492] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:50.110854 27905 catalog_manager.cc:5582] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:50.114023 28295 raft_consensus.cc:491] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:50.114538 28295 raft_consensus.cc:513] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.116540 28295 leader_election.cc:290] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:50.117192 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:50.118053 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:58:50.118265 27973 leader_election.cc:343] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:50.118772 28155 raft_consensus.cc:2466] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 0.
I20250905 19:58:50.119742 27975 leader_election.cc:304] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:50.120532 28295 raft_consensus.cc:2802] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:50.120877 28295 raft_consensus.cc:491] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:50.121191 28295 raft_consensus.cc:3058] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:50.123235 28201 tablet_bootstrap.cc:654] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:50.124380 28202 tablet_bootstrap.cc:492] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:50.124881 28202 ts_tablet_manager.cc:1397] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.057s user 0.013s sys 0.006s
I20250905 19:58:50.127597 28202 raft_consensus.cc:357] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.128290 28202 raft_consensus.cc:383] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:50.128558 28202 raft_consensus.cc:738] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:50.129194 28202 consensus_queue.cc:260] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.129364 28295 raft_consensus.cc:513] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.130267 28200 tablet_bootstrap.cc:654] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:50.130117 28276 raft_consensus.cc:491] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:50.132829 28276 raft_consensus.cc:513] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.135507 28202 ts_tablet_manager.cc:1428] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.010s user 0.004s sys 0.002s
I20250905 19:58:50.136777 28202 tablet_bootstrap.cc:492] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:50.137070 28276 leader_election.cc:290] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:50.136555 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:58:50.139175 28050 leader_election.cc:343] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:50.140059 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:50.140710 28006 raft_consensus.cc:2466] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:50.142817 28202 tablet_bootstrap.cc:654] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:50.143189 28295 leader_election.cc:290] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:50.143563 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 19:58:50.145030 27973 leader_election.cc:343] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:50.147585 28051 leader_election.cc:304] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:58:50.148571 28282 raft_consensus.cc:2802] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:50.149111 28282 raft_consensus.cc:491] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:50.149535 28282 raft_consensus.cc:3058] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:50.149801 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:50.150511 28155 raft_consensus.cc:3058] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:50.154985 28201 tablet_bootstrap.cc:492] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:50.155484 28201 ts_tablet_manager.cc:1397] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.012s
I20250905 19:58:50.158265 28201 raft_consensus.cc:357] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.159008 28201 raft_consensus.cc:383] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:50.159293 28201 raft_consensus.cc:738] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:50.160394 28282 raft_consensus.cc:513] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.161096 28202 tablet_bootstrap.cc:492] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:50.161572 28202 ts_tablet_manager.cc:1397] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.002s
I20250905 19:58:50.160497 28201 consensus_queue.cc:260] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.162747 28155 raft_consensus.cc:2466] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:58:50.163813 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:50.164409 28006 raft_consensus.cc:3058] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:50.164203 28202 raft_consensus.cc:357] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.164901 28202 raft_consensus.cc:383] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:50.165213 28202 raft_consensus.cc:738] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:50.165853 28202 consensus_queue.cc:260] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.166909 28201 ts_tablet_manager.cc:1428] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20250905 19:58:50.167861 28201 tablet_bootstrap.cc:492] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:50.168084 28202 ts_tablet_manager.cc:1428] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250905 19:58:50.170544 28200 tablet_bootstrap.cc:492] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:50.171073 28200 ts_tablet_manager.cc:1397] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.070s user 0.016s sys 0.008s
I20250905 19:58:50.171262 27975 leader_election.cc:304] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:50.172283 28295 raft_consensus.cc:2802] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:50.172744 28295 raft_consensus.cc:695] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:58:50.173746 28201 tablet_bootstrap.cc:654] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:50.173578 28295 consensus_queue.cc:237] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.173851 28200 raft_consensus.cc:357] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.174552 28200 raft_consensus.cc:383] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:50.174825 28200 raft_consensus.cc:738] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:50.175520 28200 consensus_queue.cc:260] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.183588 28200 ts_tablet_manager.cc:1428] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20250905 19:58:50.184605 28200 tablet_bootstrap.cc:492] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:50.186304 28006 raft_consensus.cc:2466] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:50.188803 28282 leader_election.cc:290] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:50.188529 28051 leader_election.cc:304] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:50.191468 28282 raft_consensus.cc:2802] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:50.191623 28200 tablet_bootstrap.cc:654] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:50.192130 28282 raft_consensus.cc:695] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:58:50.192384 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 19:58:50.197527 28050 leader_election.cc:343] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:50.205330 27905 catalog_manager.cc:5582] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:50.193249 28282 consensus_queue.cc:237] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.213621 28295 raft_consensus.cc:491] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:50.214152 28200 tablet_bootstrap.cc:492] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:50.214792 28200 ts_tablet_manager.cc:1397] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.030s user 0.016s sys 0.002s
I20250905 19:58:50.214231 28295 raft_consensus.cc:513] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.216675 28295 leader_election.cc:290] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:50.217470 28200 raft_consensus.cc:357] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.218194 28200 raft_consensus.cc:383] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:50.219341 28200 raft_consensus.cc:738] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:50.219659 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:50.220181 28200 consensus_queue.cc:260] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:58:50.221947 27973 leader_election.cc:343] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:50.228636 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:58:50.230597 27975 leader_election.cc:343] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:50.230409 28276 consensus_queue.cc:1035] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:50.231288 27975 leader_election.cc:304] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:50.232540 28295 raft_consensus.cc:2747] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:50.234869 28202 tablet_bootstrap.cc:492] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:50.235884 28200 ts_tablet_manager.cc:1428] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.021s user 0.008s sys 0.000s
I20250905 19:58:50.237304 28200 tablet_bootstrap.cc:492] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:50.243417 28295 raft_consensus.cc:491] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:50.245759 28200 tablet_bootstrap.cc:654] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:50.244154 28295 raft_consensus.cc:513] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.243795 28202 tablet_bootstrap.cc:654] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:50.250499 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:50.251988 28295 leader_election.cc:290] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:50.243676 28201 tablet_bootstrap.cc:492] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:50.252933 28201 ts_tablet_manager.cc:1397] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.085s user 0.040s sys 0.016s
W20250905 19:58:50.255167 27973 leader_election.cc:343] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:50.257153 28201 raft_consensus.cc:357] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.258034 28201 raft_consensus.cc:383] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:50.258391 28201 raft_consensus.cc:738] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:50.259305 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:50.259198 28201 consensus_queue.cc:260] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.260437 28200 tablet_bootstrap.cc:492] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
W20250905 19:58:50.260725 27975 leader_election.cc:343] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:50.260941 28200 ts_tablet_manager.cc:1397] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.011s
I20250905 19:58:50.261343 27975 leader_election.cc:304] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:50.263780 28200 raft_consensus.cc:357] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.264320 28241 raft_consensus.cc:2747] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:50.264644 28202 tablet_bootstrap.cc:492] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:50.265300 28202 ts_tablet_manager.cc:1397] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.031s user 0.014s sys 0.001s
I20250905 19:58:50.264644 28200 raft_consensus.cc:383] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:50.265960 28200 raft_consensus.cc:738] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:50.266680 28200 consensus_queue.cc:260] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.268901 28200 ts_tablet_manager.cc:1428] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.007s user 0.002s sys 0.002s
I20250905 19:58:50.269951 28200 tablet_bootstrap.cc:492] T 361ffd5c9a994e0397e6748827466d3d P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:50.276063 28200 tablet_bootstrap.cc:654] T 361ffd5c9a994e0397e6748827466d3d P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:50.276618 28201 ts_tablet_manager.cc:1428] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20250905 19:58:50.277524 27907 catalog_manager.cc:5582] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:50.268345 28202 raft_consensus.cc:357] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.279568 28202 raft_consensus.cc:383] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:50.280021 28202 raft_consensus.cc:738] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:50.285241 28201 tablet_bootstrap.cc:492] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:50.287833 28241 raft_consensus.cc:491] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:50.288404 28241 raft_consensus.cc:513] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.290544 28241 leader_election.cc:290] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:50.280761 28202 consensus_queue.cc:260] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.294152 28200 tablet_bootstrap.cc:492] T 361ffd5c9a994e0397e6748827466d3d P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:50.294725 28200 ts_tablet_manager.cc:1397] T 361ffd5c9a994e0397e6748827466d3d P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.003s
I20250905 19:58:50.294998 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:50.295909 28084 raft_consensus.cc:2466] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 0.
I20250905 19:58:50.297142 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:50.297353 28200 raft_consensus.cc:357] T 361ffd5c9a994e0397e6748827466d3d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.298081 28200 raft_consensus.cc:383] T 361ffd5c9a994e0397e6748827466d3d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:50.298358 28200 raft_consensus.cc:738] T 361ffd5c9a994e0397e6748827466d3d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:50.298795 27973 leader_election.cc:304] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:50.299068 28200 consensus_queue.cc:260] T 361ffd5c9a994e0397e6748827466d3d P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:58:50.300117 27975 leader_election.cc:343] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:50.300567 28201 tablet_bootstrap.cc:654] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:50.301324 28200 ts_tablet_manager.cc:1428] T 361ffd5c9a994e0397e6748827466d3d P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250905 19:58:50.301499 28241 raft_consensus.cc:2802] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:50.301990 28241 raft_consensus.cc:491] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:50.302302 28200 tablet_bootstrap.cc:492] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:50.302374 28241 raft_consensus.cc:3058] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:50.297389 28202 ts_tablet_manager.cc:1428] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.032s user 0.006s sys 0.001s
I20250905 19:58:50.304833 28202 tablet_bootstrap.cc:492] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:50.308779 28200 tablet_bootstrap.cc:654] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:50.316668 28276 consensus_queue.cc:1035] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:50.319051 28202 tablet_bootstrap.cc:654] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:50.324523 28201 tablet_bootstrap.cc:492] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:50.325196 28201 ts_tablet_manager.cc:1397] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.040s user 0.011s sys 0.001s
I20250905 19:58:50.328264 28201 raft_consensus.cc:357] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.329030 28201 raft_consensus.cc:383] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:50.329293 28201 raft_consensus.cc:738] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:50.330305 28201 consensus_queue.cc:260] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.333236 28237 raft_consensus.cc:491] T 361ffd5c9a994e0397e6748827466d3d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:50.333986 28237 raft_consensus.cc:513] T 361ffd5c9a994e0397e6748827466d3d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.434198 28209 raft_consensus.cc:491] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:50.440654 28209 raft_consensus.cc:513] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.445024 28206 raft_consensus.cc:491] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:58:50.433596 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 97623.2us
I20250905 19:58:50.445593 28206 raft_consensus.cc:513] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.449677 28201 ts_tablet_manager.cc:1428] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.124s user 0.006s sys 0.001s
I20250905 19:58:50.450321 28241 raft_consensus.cc:513] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.537174 28237 leader_election.cc:290] T 361ffd5c9a994e0397e6748827466d3d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:50.539608 28201 tablet_bootstrap.cc:492] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:50.541296 28237 raft_consensus.cc:491] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:50.541844 28237 raft_consensus.cc:513] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.542227 28209 leader_election.cc:290] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:50.544822 28276 raft_consensus.cc:491] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:58:50.543931 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 94448.5us
I20250905 19:58:50.549360 28209 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:50.577883 28202 tablet_bootstrap.cc:492] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:50.578656 28202 ts_tablet_manager.cc:1397] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.274s user 0.012s sys 0.002s
I20250905 19:58:50.565541 28276 raft_consensus.cc:513] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.582527 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:50.583474 28006 raft_consensus.cc:2466] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:50.587716 28200 tablet_bootstrap.cc:492] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:50.588095 28200 ts_tablet_manager.cc:1397] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.286s user 0.053s sys 0.120s
I20250905 19:58:50.584581 28276 leader_election.cc:290] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:50.573017 28209 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.594122 28237 leader_election.cc:290] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:50.600209 28241 leader_election.cc:290] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:50.601755 28202 raft_consensus.cc:357] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.596401 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:50.596400 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:50.602169 28200 raft_consensus.cc:357] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.604158 28200 raft_consensus.cc:383] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:50.604506 28200 raft_consensus.cc:738] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:50.605239 28200 consensus_queue.cc:260] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.603129 28202 raft_consensus.cc:383] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:50.606212 28202 raft_consensus.cc:738] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:50.606842 28202 consensus_queue.cc:260] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.609282 28202 ts_tablet_manager.cc:1428] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.030s user 0.005s sys 0.001s
I20250905 19:58:50.610054 28202 tablet_bootstrap.cc:492] T 18ed409dabfc45b1843e7d8a316a047d P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:50.611721 28200 ts_tablet_manager.cc:1428] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.023s user 0.006s sys 0.002s
I20250905 19:58:50.603646 28206 leader_election.cc:290] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:50.607127 28368 raft_consensus.cc:491] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:50.605118 28155 raft_consensus.cc:2466] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:50.616420 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:50.617244 28156 raft_consensus.cc:2391] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 1: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:58:50.619690 28083 raft_consensus.cc:3058] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:50.622529 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:50.628904 28366 raft_consensus.cc:491] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:50.629047 28082 raft_consensus.cc:2466] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 0.
I20250905 19:58:50.620445 28368 raft_consensus.cc:513] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.629580 28366 raft_consensus.cc:513] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.628978 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:50.630990 28010 raft_consensus.cc:2391] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 1: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:58:50.632402 28366 leader_election.cc:290] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:50.640275 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:50.640731 28156 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:50.643239 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:50.644140 28368 leader_election.cc:290] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:50.648231 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:50.650029 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:50.624410 28209 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:50.652772 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:50.656442 28202 tablet_bootstrap.cc:654] T 18ed409dabfc45b1843e7d8a316a047d P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:50.657133 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:50.682866 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:50.683597 28010 raft_consensus.cc:2466] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 0.
I20250905 19:58:50.724073 28200 tablet_bootstrap.cc:492] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:50.768958 28300 raft_consensus.cc:491] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:50.775652 28300 raft_consensus.cc:513] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:50.777776 28300 leader_election.cc:290] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:50.844568 28051 leader_election.cc:304] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:51.140856 28201 tablet_bootstrap.cc:654] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:51.142347 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:51.094317 28209 raft_consensus.cc:491] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:51.151744 28209 raft_consensus.cc:513] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.154007 28209 leader_election.cc:290] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:51.067082 28312 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:51.154839 28312 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.155377 28300 consensus_queue.cc:1035] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:51.156874 28312 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 19:58:51.160121 27956 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20250905 19:58:50.959499 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 113167us
W20250905 19:58:51.165773 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 392766us
I20250905 19:58:51.164004 28122 leader_election.cc:304] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:51.178220 28312 raft_consensus.cc:2802] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:51.174999 28369 raft_consensus.cc:2802] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:51.182173 28312 raft_consensus.cc:491] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:51.183128 28369 raft_consensus.cc:491] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:51.185411 28312 raft_consensus.cc:3058] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:51.183607 28125 leader_election.cc:304] T 361ffd5c9a994e0397e6748827466d3d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:51.189294 28200 tablet_bootstrap.cc:654] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
W20250905 19:58:51.193712 28125 leader_election.cc:343] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:51.196172 28312 raft_consensus.cc:513] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.198083 28312 leader_election.cc:290] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 19:58:51.200016 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 53436.8us
I20250905 19:58:51.204840 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:51.206600 28154 raft_consensus.cc:3058] T 361ffd5c9a994e0397e6748827466d3d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:51.185827 28369 raft_consensus.cc:3058] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:51.200842 28050 leader_election.cc:304] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:51.219250 28370 raft_consensus.cc:3058] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:51.220562 28201 tablet_bootstrap.cc:492] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:51.221076 28201 ts_tablet_manager.cc:1397] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.682s user 0.012s sys 0.011s
W20250905 19:58:51.226732 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 45494.8us
W20250905 19:58:51.228783 28122 leader_election.cc:343] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:51.231077 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:51.240260 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:51.241766 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:51.240176 28370 raft_consensus.cc:2747] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:51.259824 28159 raft_consensus.cc:2391] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 1: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:58:51.240095 28084 raft_consensus.cc:3058] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:58:51.243896 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 95788.6us
I20250905 19:58:51.246162 28201 raft_consensus.cc:357] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.253119 28200 tablet_bootstrap.cc:492] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:51.262132 28201 raft_consensus.cc:383] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:51.262722 28201 raft_consensus.cc:738] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:51.262269 28200 ts_tablet_manager.cc:1397] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.538s user 0.016s sys 0.007s
I20250905 19:58:51.246558 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:58:51.257814 28373 raft_consensus.cc:2747] T 361ffd5c9a994e0397e6748827466d3d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:51.264267 28201 consensus_queue.cc:260] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:58:51.244515 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 94161.7us
I20250905 19:58:51.269251 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:51.269666 28050 leader_election.cc:304] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:51.269274 28368 consensus_queue.cc:1035] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250905 19:58:51.270175 28200 raft_consensus.cc:357] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.271258 28200 raft_consensus.cc:383] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:51.271554 28200 raft_consensus.cc:738] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:51.272248 28200 consensus_queue.cc:260] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.263707 28082 raft_consensus.cc:3058] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:58:51.275481 28050 leader_election.cc:343] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:51.278151 28050 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:51.279228 28372 raft_consensus.cc:2747] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:51.261793 28122 leader_election.cc:304] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
W20250905 19:58:51.284058 27975 leader_election.cc:343] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:51.284094 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:51.284833 28366 raft_consensus.cc:2747] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:51.284936 28009 raft_consensus.cc:2466] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:51.286484 28200 ts_tablet_manager.cc:1428] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20250905 19:58:51.287241 28202 tablet_bootstrap.cc:492] T 18ed409dabfc45b1843e7d8a316a047d P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:51.287866 28202 ts_tablet_manager.cc:1397] T 18ed409dabfc45b1843e7d8a316a047d P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.678s user 0.060s sys 0.109s
I20250905 19:58:51.288071 28200 tablet_bootstrap.cc:492] T 2bd6bb972f14472a89ccf99de880efc2 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
W20250905 19:58:51.287823 28125 leader_election.cc:343] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:51.242987 28369 raft_consensus.cc:513] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.289477 28201 ts_tablet_manager.cc:1428] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.068s user 0.006s sys 0.004s
I20250905 19:58:51.288425 28368 raft_consensus.cc:2747] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:51.299397 28201 tablet_bootstrap.cc:492] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
W20250905 19:58:51.284209 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 55495.2us
I20250905 19:58:51.304241 28202 raft_consensus.cc:357] T 18ed409dabfc45b1843e7d8a316a047d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.306555 28202 raft_consensus.cc:383] T 18ed409dabfc45b1843e7d8a316a047d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:51.306998 28202 raft_consensus.cc:738] T 18ed409dabfc45b1843e7d8a316a047d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:51.307952 28202 consensus_queue.cc:260] T 18ed409dabfc45b1843e7d8a316a047d P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.317687 28298 consensus_queue.cc:1035] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:51.304598 28082 raft_consensus.cc:2466] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
W20250905 19:58:51.324852 28122 leader_election.cc:343] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:51.329026 28202 ts_tablet_manager.cc:1428] T 18ed409dabfc45b1843e7d8a316a047d P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.041s user 0.008s sys 0.002s
I20250905 19:58:51.330513 28202 tablet_bootstrap.cc:492] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:51.335996 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:51.336215 28122 leader_election.cc:304] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:51.336799 28084 raft_consensus.cc:2466] T 18ed409dabfc45b1843e7d8a316a047d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 0.
I20250905 19:58:51.337381 28366 raft_consensus.cc:2747] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:51.344517 28369 leader_election.cc:290] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:51.347728 28376 consensus_queue.cc:1035] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250905 19:58:51.350227 27975 leader_election.cc:343] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:51.364027 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:51.371626 28369 consensus_queue.cc:1035] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:51.382876 28376 consensus_queue.cc:1035] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:51.384876 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:51.385957 28009 raft_consensus.cc:3058] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:51.390363 28200 tablet_bootstrap.cc:654] T 2bd6bb972f14472a89ccf99de880efc2 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:51.390197 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:51.391059 28154 raft_consensus.cc:2391] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 1: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:58:51.392251 28376 consensus_queue.cc:1035] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:51.393294 28051 leader_election.cc:304] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:58:51.394162 28368 raft_consensus.cc:2802] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:51.394956 28368 raft_consensus.cc:491] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:51.395321 28368 raft_consensus.cc:3058] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:51.398244 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:51.401582 28202 tablet_bootstrap.cc:654] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:51.402815 28201 tablet_bootstrap.cc:654] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:51.400983 28009 raft_consensus.cc:2466] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 19:58:51.406163 28368 raft_consensus.cc:513] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.417697 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:51.418710 28154 raft_consensus.cc:3058] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:51.425697 28200 tablet_bootstrap.cc:492] T 2bd6bb972f14472a89ccf99de880efc2 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:51.426489 28200 ts_tablet_manager.cc:1397] T 2bd6bb972f14472a89ccf99de880efc2 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.139s user 0.019s sys 0.009s
I20250905 19:58:51.434146 28200 raft_consensus.cc:357] T 2bd6bb972f14472a89ccf99de880efc2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.435102 28200 raft_consensus.cc:383] T 2bd6bb972f14472a89ccf99de880efc2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:51.435541 28200 raft_consensus.cc:738] T 2bd6bb972f14472a89ccf99de880efc2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:51.422286 28368 leader_election.cc:290] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:51.436506 28200 consensus_queue.cc:260] T 2bd6bb972f14472a89ccf99de880efc2 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.413089 27973 leader_election.cc:304] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:58:51.439180 28200 ts_tablet_manager.cc:1428] T 2bd6bb972f14472a89ccf99de880efc2 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20250905 19:58:51.441233 28200 tablet_bootstrap.cc:492] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:51.441915 28377 raft_consensus.cc:2802] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:51.442364 28377 raft_consensus.cc:491] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:51.442716 28377 raft_consensus.cc:3058] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:51.445572 28154 raft_consensus.cc:2466] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:51.449678 28050 leader_election.cc:304] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:51.450785 28368 raft_consensus.cc:2802] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:51.451685 28201 tablet_bootstrap.cc:492] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:51.452107 28201 ts_tablet_manager.cc:1397] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.153s user 0.012s sys 0.006s
I20250905 19:58:51.451362 28368 raft_consensus.cc:695] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:58:51.463577 28368 consensus_queue.cc:237] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.481531 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:51.482290 28085 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 19:58:51.478171 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:58:51.486604 28082 raft_consensus.cc:2391] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 1: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:58:51.490227 28201 raft_consensus.cc:357] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.491139 28201 raft_consensus.cc:383] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:51.491426 28201 raft_consensus.cc:738] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:51.492134 28201 consensus_queue.cc:260] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.480981 27973 leader_election.cc:304] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:58:51.495487 28376 raft_consensus.cc:2802] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:51.495980 28376 raft_consensus.cc:695] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:58:51.496845 28376 consensus_queue.cc:237] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.497159 28377 raft_consensus.cc:513] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.505684 28202 tablet_bootstrap.cc:492] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:51.506379 28202 ts_tablet_manager.cc:1397] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.176s user 0.015s sys 0.017s
I20250905 19:58:51.508596 28200 tablet_bootstrap.cc:654] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:51.509999 28201 ts_tablet_manager.cc:1428] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.058s user 0.007s sys 0.000s
I20250905 19:58:51.511059 28201 tablet_bootstrap.cc:492] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:51.515231 28202 raft_consensus.cc:357] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.516503 28202 raft_consensus.cc:383] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:51.518736 28202 raft_consensus.cc:738] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:51.521696 28300 consensus_queue.cc:1035] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:51.522699 28377 leader_election.cc:290] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:51.525399 28202 consensus_queue.cc:260] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.535346 28202 ts_tablet_manager.cc:1428] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.029s user 0.006s sys 0.004s
I20250905 19:58:51.536440 28202 tablet_bootstrap.cc:492] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
W20250905 19:58:51.538034 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 103237us
I20250905 19:58:51.517843 28122 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:51.554013 28377 raft_consensus.cc:491] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:51.559337 28377 raft_consensus.cc:513] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:58:51.559804 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 47142.9us
I20250905 19:58:51.561600 28377 leader_election.cc:290] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:51.565065 28369 raft_consensus.cc:491] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:51.565764 28369 raft_consensus.cc:513] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.561641 28312 raft_consensus.cc:2802] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:58:51.567477 28312 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:58:51.567670 28369 leader_election.cc:290] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:51.567911 28312 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:51.582954 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:51.583871 28006 raft_consensus.cc:3058] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:51.594460 28298 raft_consensus.cc:491] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:51.595295 28298 raft_consensus.cc:513] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.598356 28298 leader_election.cc:290] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:58:51.607461 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 76755.8us
I20250905 19:58:51.613152 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:51.614053 28082 raft_consensus.cc:2466] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 0.
I20250905 19:58:51.614197 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:51.617974 28157 raft_consensus.cc:2466] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
W20250905 19:58:51.618191 28391 log.cc:927] Time spent T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20250905 19:58:51.614648 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:51.614836 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:51.616570 28201 tablet_bootstrap.cc:654] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:51.618248 27973 leader_election.cc:304] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:51.620175 28125 leader_election.cc:304] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:51.576066 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:58:51.619863 28050 leader_election.cc:304] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:51.621240 28298 raft_consensus.cc:2802] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:51.624253 28368 raft_consensus.cc:2802] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:51.624570 28298 raft_consensus.cc:491] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:51.625070 28298 raft_consensus.cc:3058] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:51.624614 27905 catalog_manager.cc:5582] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:51.623355 28366 raft_consensus.cc:2802] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:51.631012 28366 raft_consensus.cc:695] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:58:51.631918 28366 consensus_queue.cc:237] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:58:51.632642 27973 leader_election.cc:343] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:51.628741 28083 raft_consensus.cc:3058] T 18ed409dabfc45b1843e7d8a316a047d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:51.624646 28368 raft_consensus.cc:491] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:51.622295 28051 leader_election.cc:304] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:51.643143 28368 raft_consensus.cc:3058] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:58:51.660503 28392 log.cc:927] Time spent T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.096s user 0.001s sys 0.000s
I20250905 19:58:51.663748 28370 raft_consensus.cc:2747] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250905 19:58:51.666848 28202 tablet_bootstrap.cc:654] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:51.684554 28373 raft_consensus.cc:491] T 2bd6bb972f14472a89ccf99de880efc2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:51.685350 28373 raft_consensus.cc:513] T 2bd6bb972f14472a89ccf99de880efc2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:58:51.699083 28394 log.cc:927] Time spent T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.149s user 0.004s sys 0.000s
W20250905 19:58:51.702788 28396 log.cc:927] Time spent T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20250905 19:58:51.727442 27907 catalog_manager.cc:5582] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:51.767756 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:51.734061 28368 raft_consensus.cc:513] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.772034 28368 leader_election.cc:290] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:58:51.750977 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 169313us
I20250905 19:58:51.770095 28083 raft_consensus.cc:2466] T 18ed409dabfc45b1843e7d8a316a047d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
W20250905 19:58:51.776716 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 149230us
I20250905 19:58:51.783212 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:51.784010 28159 raft_consensus.cc:3058] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:51.787911 27908 catalog_manager.cc:5582] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
W20250905 19:58:51.788784 27975 leader_election.cc:343] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:51.793486 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:51.798249 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:51.799773 28373 leader_election.cc:290] T 2bd6bb972f14472a89ccf99de880efc2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:51.796427 28006 raft_consensus.cc:2466] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:51.809791 28200 tablet_bootstrap.cc:492] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:51.810505 28200 ts_tablet_manager.cc:1397] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.370s user 0.013s sys 0.016s
W20250905 19:58:51.813105 27975 leader_election.cc:343] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:51.815732 27975 leader_election.cc:304] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:51.813818 27973 leader_election.cc:304] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:58:51.820008 28300 raft_consensus.cc:2747] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:51.820642 28300 raft_consensus.cc:2802] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:51.821118 28300 raft_consensus.cc:695] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:58:51.822212 28300 consensus_queue.cc:237] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.812857 28312 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.829916 28154 raft_consensus.cc:1238] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 1. Current term is 2. Ops: []
I20250905 19:58:51.830992 28312 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:51.848838 28298 raft_consensus.cc:513] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.851683 28298 leader_election.cc:290] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:58:51.814512 28403 log.cc:927] Time spent T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20250905 19:58:51.862910 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:58:51.863824 28081 raft_consensus.cc:2391] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 1: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:58:51.865978 28200 raft_consensus.cc:357] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.866918 28200 raft_consensus.cc:383] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:51.867211 28200 raft_consensus.cc:738] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:51.868026 28200 consensus_queue.cc:260] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.868037 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:51.869854 28009 raft_consensus.cc:2391] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 1: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:58:51.870147 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:51.870661 28200 ts_tablet_manager.cc:1428] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.060s user 0.010s sys 0.003s
I20250905 19:58:51.870981 28010 raft_consensus.cc:2391] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 1: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:58:51.871682 28200 tablet_bootstrap.cc:492] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:51.862952 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:51.876677 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:51.877542 28201 tablet_bootstrap.cc:492] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:51.877673 28081 raft_consensus.cc:2391] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 1: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:58:51.878073 28201 ts_tablet_manager.cc:1397] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.367s user 0.010s sys 0.025s
I20250905 19:58:51.879760 28159 raft_consensus.cc:2466] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:51.881027 28201 raft_consensus.cc:357] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.881325 28050 leader_election.cc:304] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:51.881763 28201 raft_consensus.cc:383] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:51.882086 28201 raft_consensus.cc:738] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:51.882867 28201 consensus_queue.cc:260] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.893074 28368 raft_consensus.cc:2802] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:51.896332 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:58:51.894147 28368 raft_consensus.cc:695] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:58:51.897635 28201 ts_tablet_manager.cc:1428] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.019s user 0.009s sys 0.000s
I20250905 19:58:51.897866 28202 tablet_bootstrap.cc:492] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:51.898300 28202 ts_tablet_manager.cc:1397] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.362s user 0.013s sys 0.008s
I20250905 19:58:51.898733 28201 tablet_bootstrap.cc:492] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:51.904067 28202 raft_consensus.cc:357] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.904795 28202 raft_consensus.cc:383] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:51.905092 28202 raft_consensus.cc:738] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:51.906785 28404 consensus_queue.cc:1046] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.002s
I20250905 19:58:51.907976 28404 raft_consensus.cc:3053] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:58:51.908378 28404 raft_consensus.cc:738] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:58:51.909201 28404 consensus_queue.cc:260] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.905351 28122 leader_election.cc:304] T 2bd6bb972f14472a89ccf99de880efc2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:51.912192 28373 raft_consensus.cc:3058] T 2bd6bb972f14472a89ccf99de880efc2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:51.914885 28122 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:51.916013 28312 raft_consensus.cc:2747] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 19:58:51.898408 28368 consensus_queue.cc:237] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.919188 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:51.919090 28202 consensus_queue.cc:260] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.920092 28159 raft_consensus.cc:2391] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 1: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:58:51.922466 28202 ts_tablet_manager.cc:1428] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.022s user 0.011s sys 0.006s
I20250905 19:58:51.923491 28202 tablet_bootstrap.cc:492] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:51.919216 28404 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:58:51.925225 28373 raft_consensus.cc:2747] T 2bd6bb972f14472a89ccf99de880efc2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:51.928797 28369 consensus_queue.cc:1035] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:51.929409 28200 tablet_bootstrap.cc:654] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:51.949141 28201 tablet_bootstrap.cc:654] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:51.949131 28202 tablet_bootstrap.cc:654] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:51.954897 27975 leader_election.cc:304] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:51.958079 28404 raft_consensus.cc:2747] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250905 19:58:51.959012 28300 consensus_queue.cc:1035] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:51.964555 28369 consensus_queue.cc:1035] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:51.966490 28201 tablet_bootstrap.cc:492] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:51.967262 28201 ts_tablet_manager.cc:1397] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.069s user 0.010s sys 0.005s
I20250905 19:58:51.980614 28201 raft_consensus.cc:357] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.981704 28201 raft_consensus.cc:383] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:51.982081 28201 raft_consensus.cc:738] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:51.985978 28200 tablet_bootstrap.cc:492] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:51.990805 28200 ts_tablet_manager.cc:1397] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.119s user 0.013s sys 0.002s
I20250905 19:58:51.989689 28202 tablet_bootstrap.cc:492] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:51.992185 28202 ts_tablet_manager.cc:1397] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.069s user 0.018s sys 0.020s
I20250905 19:58:51.995016 28393 consensus_queue.cc:1035] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:51.997584 28202 raft_consensus.cc:357] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.998613 28202 raft_consensus.cc:383] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:51.999137 28202 raft_consensus.cc:738] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:52.000005 28202 consensus_queue.cc:260] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.995103 28200 raft_consensus.cc:357] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.001619 28200 raft_consensus.cc:383] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:52.002553 28200 raft_consensus.cc:738] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:52.004065 28200 consensus_queue.cc:260] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:51.986732 28201 consensus_queue.cc:260] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.012166 28201 ts_tablet_manager.cc:1428] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.045s user 0.009s sys 0.000s
I20250905 19:58:52.013309 28201 tablet_bootstrap.cc:492] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:52.016234 28202 ts_tablet_manager.cc:1428] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.024s user 0.009s sys 0.000s
I20250905 19:58:52.017359 28202 tablet_bootstrap.cc:492] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:52.017422 28300 raft_consensus.cc:491] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:52.018654 28300 raft_consensus.cc:513] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.022869 28200 ts_tablet_manager.cc:1428] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.031s user 0.006s sys 0.001s
I20250905 19:58:52.025795 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:52.026549 28159 raft_consensus.cc:2466] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 0.
I20250905 19:58:52.027701 27975 leader_election.cc:304] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:58:52.028721 28393 raft_consensus.cc:2802] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:52.028781 28373 consensus_queue.cc:1035] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:52.029143 28393 raft_consensus.cc:491] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:52.029510 28300 leader_election.cc:290] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:52.031155 28200 tablet_bootstrap.cc:492] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:52.026245 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:52.032130 28393 raft_consensus.cc:3058] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:52.048523 27908 catalog_manager.cc:5582] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
W20250905 19:58:52.052096 27973 leader_election.cc:343] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:52.057482 28393 raft_consensus.cc:513] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.062058 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 19:58:52.065371 27973 leader_election.cc:343] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:52.066388 28393 leader_election.cc:290] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:52.067957 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:52.068606 28159 raft_consensus.cc:3058] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:52.073964 28202 tablet_bootstrap.cc:654] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:52.082489 28200 tablet_bootstrap.cc:654] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:52.085883 28201 tablet_bootstrap.cc:654] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:52.089174 28159 raft_consensus.cc:2466] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:58:52.091082 27975 leader_election.cc:304] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:52.092180 28393 raft_consensus.cc:2802] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:52.092798 28393 raft_consensus.cc:695] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:58:52.093701 28393 consensus_queue.cc:237] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.096251 27905 catalog_manager.cc:5582] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:52.124071 28366 consensus_queue.cc:1035] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:52.135434 28202 tablet_bootstrap.cc:492] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:52.136056 28202 ts_tablet_manager.cc:1397] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.119s user 0.021s sys 0.017s
I20250905 19:58:52.140085 28202 raft_consensus.cc:357] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.140920 28202 raft_consensus.cc:383] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:52.141192 28202 raft_consensus.cc:738] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:52.142135 28201 tablet_bootstrap.cc:492] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:52.141855 28202 consensus_queue.cc:260] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.142673 28201 ts_tablet_manager.cc:1397] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.130s user 0.010s sys 0.007s
I20250905 19:58:52.143819 28200 tablet_bootstrap.cc:492] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:52.144311 28200 ts_tablet_manager.cc:1397] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.113s user 0.027s sys 0.019s
I20250905 19:58:52.144434 28202 ts_tablet_manager.cc:1428] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250905 19:58:52.145483 28202 tablet_bootstrap.cc:492] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:52.145828 28201 raft_consensus.cc:357] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.146633 28201 raft_consensus.cc:383] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:52.146907 28369 raft_consensus.cc:491] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:52.146984 28201 raft_consensus.cc:738] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:52.146804 28200 raft_consensus.cc:357] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.147631 28200 raft_consensus.cc:383] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:52.147519 28369 raft_consensus.cc:513] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.147953 28200 raft_consensus.cc:738] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:52.147987 28201 consensus_queue.cc:260] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.148659 28200 consensus_queue.cc:260] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.149565 28369 leader_election.cc:290] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:52.150910 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:52.152802 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:52.153540 28009 raft_consensus.cc:2466] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:52.154330 28393 raft_consensus.cc:491] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:52.154853 28393 raft_consensus.cc:513] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.156055 28202 tablet_bootstrap.cc:654] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:52.156931 28393 leader_election.cc:290] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:58:52.158593 28050 leader_election.cc:343] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:52.160631 28051 leader_election.cc:304] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:58:52.162312 28369 raft_consensus.cc:2802] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:52.162811 28369 raft_consensus.cc:491] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:52.163147 28369 raft_consensus.cc:3058] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:52.165071 28200 ts_tablet_manager.cc:1428] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.020s user 0.006s sys 0.001s
I20250905 19:58:52.166224 28200 tablet_bootstrap.cc:492] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:52.173246 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:52.178509 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:58:52.178972 27975 leader_election.cc:343] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:52.181841 27907 catalog_manager.cc:5582] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
W20250905 19:58:52.183616 27973 leader_election.cc:343] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:52.184115 27973 leader_election.cc:304] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:52.186182 28300 raft_consensus.cc:2747] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:52.195945 28201 ts_tablet_manager.cc:1428] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.053s user 0.004s sys 0.002s
I20250905 19:58:52.197448 28201 tablet_bootstrap.cc:492] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:52.200172 28369 raft_consensus.cc:513] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.204979 28368 raft_consensus.cc:491] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:52.205529 28368 raft_consensus.cc:513] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.207691 28368 leader_election.cc:290] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:52.208698 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:52.209429 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:58:52.210480 28051 leader_election.cc:343] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:58:52.211103 28050 leader_election.cc:343] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:52.211549 28050 leader_election.cc:304] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:52.214018 28368 raft_consensus.cc:2747] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:52.214569 28373 raft_consensus.cc:491] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:52.215075 28373 raft_consensus.cc:513] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.217301 28373 leader_election.cc:290] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:52.221297 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:52.221865 28159 raft_consensus.cc:3058] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:52.224007 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:52.228318 28081 raft_consensus.cc:2391] T 18ed409dabfc45b1843e7d8a316a047d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 1: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:58:52.228211 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:52.229213 28009 raft_consensus.cc:3058] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:52.230100 28122 leader_election.cc:304] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:52.237742 28201 tablet_bootstrap.cc:654] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:52.238565 28369 leader_election.cc:290] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:52.241052 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 19:58:52.245430 28050 leader_election.cc:343] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:52.258322 28366 raft_consensus.cc:491] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:52.259007 28366 raft_consensus.cc:513] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.261085 28366 leader_election.cc:290] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:52.263144 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:58:52.264698 28122 leader_election.cc:343] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:52.265101 28200 tablet_bootstrap.cc:654] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:52.265563 28369 raft_consensus.cc:491] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:52.266079 28369 raft_consensus.cc:513] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.266996 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:52.267923 28010 raft_consensus.cc:2466] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 0.
I20250905 19:58:52.268294 28369 leader_election.cc:290] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:52.269423 28125 leader_election.cc:304] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:52.272166 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:52.274160 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:58:52.274281 28051 leader_election.cc:343] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:52.275249 28154 raft_consensus.cc:2466] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:52.277868 28366 raft_consensus.cc:2802] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:52.278123 28050 leader_election.cc:304] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:52.278609 28366 raft_consensus.cc:491] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:52.279199 28366 raft_consensus.cc:3058] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:52.279399 28369 raft_consensus.cc:2802] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:52.279870 28369 raft_consensus.cc:491] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:52.280223 28369 raft_consensus.cc:3058] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:52.284102 28373 raft_consensus.cc:2747] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:52.285346 28300 raft_consensus.cc:491] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:52.285913 28300 raft_consensus.cc:513] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.288183 28300 leader_election.cc:290] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:52.288362 28393 consensus_queue.cc:1035] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:52.289664 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:52.291208 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:58:52.293183 27973 leader_election.cc:343] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:58:52.293825 27975 leader_election.cc:343] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:52.294272 27975 leader_election.cc:304] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:52.295105 28393 raft_consensus.cc:2747] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:52.295708 28369 raft_consensus.cc:513] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.297791 28369 leader_election.cc:290] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:52.299239 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:52.300074 28156 raft_consensus.cc:3058] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:52.302563 28366 raft_consensus.cc:513] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.303400 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:52.305279 28366 leader_election.cc:290] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:52.305316 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:52.306682 28006 raft_consensus.cc:3058] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:58:52.318347 28051 leader_election.cc:343] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:52.321821 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 19:58:52.323760 28122 leader_election.cc:343] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:52.336572 28006 raft_consensus.cc:2466] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 19:58:52.325603 28201 tablet_bootstrap.cc:492] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:52.338551 28201 ts_tablet_manager.cc:1397] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.141s user 0.015s sys 0.008s
I20250905 19:58:52.337937 28125 leader_election.cc:304] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:52.341910 28201 raft_consensus.cc:357] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.343071 28202 tablet_bootstrap.cc:492] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:52.347714 28201 raft_consensus.cc:383] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:52.348161 28201 raft_consensus.cc:738] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:52.348304 28202 ts_tablet_manager.cc:1397] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.203s user 0.055s sys 0.073s
I20250905 19:58:52.347469 28366 raft_consensus.cc:2802] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:52.350345 28366 raft_consensus.cc:695] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:58:52.340157 28156 raft_consensus.cc:2466] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:52.351256 28366 consensus_queue.cc:237] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.352828 28050 leader_election.cc:304] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:52.354720 28202 raft_consensus.cc:357] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.343815 28009 raft_consensus.cc:2466] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:52.355441 28202 raft_consensus.cc:383] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:52.355566 28369 raft_consensus.cc:2802] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:52.348977 28201 consensus_queue.cc:260] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.356600 28202 raft_consensus.cc:738] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:52.357580 28202 consensus_queue.cc:260] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.358891 28051 leader_election.cc:304] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:58:52.359997 28369 raft_consensus.cc:695] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:58:52.360527 28202 ts_tablet_manager.cc:1428] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20250905 19:58:52.361232 28202 tablet_bootstrap.cc:492] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:52.360886 28369 consensus_queue.cc:237] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.365897 28201 ts_tablet_manager.cc:1428] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.027s user 0.005s sys 0.001s
I20250905 19:58:52.366851 28200 tablet_bootstrap.cc:492] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:52.370116 28200 ts_tablet_manager.cc:1397] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.204s user 0.013s sys 0.003s
I20250905 19:58:52.366986 28201 tablet_bootstrap.cc:492] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:52.371498 27907 catalog_manager.cc:5582] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20250905 19:58:52.370594 28368 raft_consensus.cc:2802] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:52.373582 28368 raft_consensus.cc:695] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
W20250905 19:58:52.379856 27956 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250905 19:58:52.389212 28368 consensus_queue.cc:237] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:58:52.442015 28431 log.cc:927] Time spent T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20250905 19:58:52.448709 28200 raft_consensus.cc:357] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.451184 28200 raft_consensus.cc:383] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:52.451504 28200 raft_consensus.cc:738] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:52.452147 28200 consensus_queue.cc:260] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.454881 27908 catalog_manager.cc:5582] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:52.458626 28369 raft_consensus.cc:491] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:52.459129 28369 raft_consensus.cc:513] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.461323 28369 leader_election.cc:290] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:52.465900 28202 tablet_bootstrap.cc:654] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:52.472607 28200 ts_tablet_manager.cc:1428] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.101s user 0.013s sys 0.012s
I20250905 19:58:52.473841 28200 tablet_bootstrap.cc:492] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:52.477023 28370 consensus_queue.cc:1035] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:52.479933 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:52.480629 28008 raft_consensus.cc:2466] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:52.484246 28201 tablet_bootstrap.cc:654] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:52.493840 28051 leader_election.cc:304] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:52.496248 28436 raft_consensus.cc:2802] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:52.500114 28436 raft_consensus.cc:491] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:52.500597 28436 raft_consensus.cc:3058] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:52.501606 27908 catalog_manager.cc:5582] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:52.502985 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:52.503630 28157 raft_consensus.cc:2466] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:52.503970 28200 tablet_bootstrap.cc:654] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:52.508471 28300 consensus_queue.cc:1035] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:52.534189 28201 tablet_bootstrap.cc:492] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:52.534745 28201 ts_tablet_manager.cc:1397] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.168s user 0.035s sys 0.044s
I20250905 19:58:52.537750 28201 raft_consensus.cc:357] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.538506 28201 raft_consensus.cc:383] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:52.538770 28201 raft_consensus.cc:738] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:52.539476 28201 consensus_queue.cc:260] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.540596 28436 raft_consensus.cc:513] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.543429 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:52.544088 28006 raft_consensus.cc:3058] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:52.547222 28201 ts_tablet_manager.cc:1428] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20250905 19:58:52.548358 28201 tablet_bootstrap.cc:492] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:52.555095 28436 leader_election.cc:290] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:52.556339 28202 tablet_bootstrap.cc:492] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:52.556746 28202 ts_tablet_manager.cc:1397] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.196s user 0.013s sys 0.003s
I20250905 19:58:52.559685 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:52.560335 28157 raft_consensus.cc:3058] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:52.562649 28006 raft_consensus.cc:2466] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:52.566757 28051 leader_election.cc:304] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:52.567207 28368 consensus_queue.cc:1035] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:52.567970 28370 raft_consensus.cc:2802] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:52.568440 28370 raft_consensus.cc:695] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:58:52.568796 28300 raft_consensus.cc:491] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:52.569263 28300 raft_consensus.cc:513] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.569286 28370 consensus_queue.cc:237] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.579903 28201 tablet_bootstrap.cc:654] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:52.583425 28300 leader_election.cc:290] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:52.583508 28157 raft_consensus.cc:2466] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:52.585647 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:58:52.587445 27975 leader_election.cc:343] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:52.600261 28200 tablet_bootstrap.cc:492] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:52.600689 28200 ts_tablet_manager.cc:1397] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.127s user 0.027s sys 0.026s
I20250905 19:58:52.601536 28393 consensus_queue.cc:1035] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:52.602089 28202 raft_consensus.cc:357] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.602823 28202 raft_consensus.cc:383] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:52.603096 28202 raft_consensus.cc:738] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:52.603695 28202 consensus_queue.cc:260] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.603781 28200 raft_consensus.cc:357] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.604626 28200 raft_consensus.cc:383] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:52.604967 28200 raft_consensus.cc:738] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:52.605695 28200 consensus_queue.cc:260] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.611851 28200 ts_tablet_manager.cc:1428] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.011s user 0.005s sys 0.002s
I20250905 19:58:52.613224 28202 ts_tablet_manager.cc:1428] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.056s user 0.004s sys 0.002s
I20250905 19:58:52.614065 28202 tablet_bootstrap.cc:492] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:52.617823 28200 tablet_bootstrap.cc:492] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:52.619410 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:52.620191 28083 raft_consensus.cc:2466] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 0.
I20250905 19:58:52.627429 28200 tablet_bootstrap.cc:654] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:52.631661 28202 tablet_bootstrap.cc:654] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:52.632557 28201 tablet_bootstrap.cc:492] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:52.633185 28201 ts_tablet_manager.cc:1397] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.085s user 0.008s sys 0.003s
I20250905 19:58:52.634848 27973 leader_election.cc:304] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:58:52.636622 28393 raft_consensus.cc:2802] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:52.637030 28393 raft_consensus.cc:491] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:52.637372 28393 raft_consensus.cc:3058] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:52.652647 28370 raft_consensus.cc:491] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:52.656814 28201 raft_consensus.cc:357] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.658108 28201 raft_consensus.cc:383] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:52.657688 27908 catalog_manager.cc:5582] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:52.658496 28201 raft_consensus.cc:738] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:52.653232 28370 raft_consensus.cc:513] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.659407 28393 raft_consensus.cc:513] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.661146 28370 leader_election.cc:290] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:52.661443 28393 leader_election.cc:290] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:52.659415 28201 consensus_queue.cc:260] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.662724 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:52.663399 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:52.663640 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:52.664465 28008 raft_consensus.cc:2391] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 1: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
W20250905 19:58:52.665014 27975 leader_election.cc:343] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:52.665486 28201 ts_tablet_manager.cc:1428] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.032s user 0.009s sys 0.001s
I20250905 19:58:52.666574 28201 tablet_bootstrap.cc:492] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:52.666790 28051 leader_election.cc:304] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:52.667179 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:58:52.667780 28083 raft_consensus.cc:3058] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:52.668009 28370 raft_consensus.cc:3058] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:52.674686 28202 tablet_bootstrap.cc:492] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:52.675215 28202 ts_tablet_manager.cc:1397] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.061s user 0.012s sys 0.000s
W20250905 19:58:52.675374 27973 consensus_peers.cc:489] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 -> Peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Couldn't send request to peer e4413ae5e0b64b52a6f360ac580239b9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:58:52.676631 28083 raft_consensus.cc:2466] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:58:52.677963 27973 leader_election.cc:304] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:58:52.678944 28393 raft_consensus.cc:2802] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:52.679418 28393 raft_consensus.cc:695] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:58:52.678217 28202 raft_consensus.cc:357] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.680202 28200 tablet_bootstrap.cc:492] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:52.680506 28202 raft_consensus.cc:383] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:52.680212 28393 consensus_queue.cc:237] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.680763 28200 ts_tablet_manager.cc:1397] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.064s user 0.012s sys 0.020s
I20250905 19:58:52.680788 28202 raft_consensus.cc:738] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:52.682265 28201 tablet_bootstrap.cc:654] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:52.681876 28202 consensus_queue.cc:260] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.684296 28200 raft_consensus.cc:357] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.684815 28202 ts_tablet_manager.cc:1428] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250905 19:58:52.685047 28200 raft_consensus.cc:383] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:52.685986 28202 tablet_bootstrap.cc:492] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:52.687109 28200 raft_consensus.cc:738] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:52.688205 28200 consensus_queue.cc:260] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.691769 28200 ts_tablet_manager.cc:1428] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20250905 19:58:52.693104 28200 tablet_bootstrap.cc:492] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:52.695662 28370 raft_consensus.cc:2747] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:52.712410 28202 tablet_bootstrap.cc:654] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:52.713006 28200 tablet_bootstrap.cc:654] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:52.722114 27908 catalog_manager.cc:5582] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:52.727208 28200 tablet_bootstrap.cc:492] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:52.727705 28200 ts_tablet_manager.cc:1397] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.035s user 0.013s sys 0.004s
I20250905 19:58:52.729650 28202 tablet_bootstrap.cc:492] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:52.730263 28202 ts_tablet_manager.cc:1397] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.044s user 0.011s sys 0.000s
I20250905 19:58:52.730598 28201 tablet_bootstrap.cc:492] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:52.731103 28201 ts_tablet_manager.cc:1397] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.065s user 0.015s sys 0.034s
I20250905 19:58:52.733209 28202 raft_consensus.cc:357] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.733480 28201 raft_consensus.cc:357] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.734021 28202 raft_consensus.cc:383] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:52.734496 28202 raft_consensus.cc:738] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:52.735327 28200 raft_consensus.cc:357] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.736021 28200 raft_consensus.cc:383] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:52.736460 28202 consensus_queue.cc:260] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.737128 28200 raft_consensus.cc:738] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:52.738636 28202 ts_tablet_manager.cc:1428] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250905 19:58:52.740419 28202 tablet_bootstrap.cc:492] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:52.734206 28201 raft_consensus.cc:383] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:52.746112 28201 raft_consensus.cc:738] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:52.746356 28393 raft_consensus.cc:491] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:52.746780 28201 consensus_queue.cc:260] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.746882 28393 raft_consensus.cc:513] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.748901 28393 leader_election.cc:290] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:52.749022 28201 ts_tablet_manager.cc:1428] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.018s user 0.003s sys 0.001s
I20250905 19:58:52.749670 28200 consensus_queue.cc:260] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.751132 28201 tablet_bootstrap.cc:492] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:52.751996 28202 tablet_bootstrap.cc:654] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:52.752038 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:52.754819 28375 raft_consensus.cc:491] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:52.755324 28375 raft_consensus.cc:513] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:58:52.756107 27973 leader_election.cc:343] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:52.757228 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:52.757385 28375 leader_election.cc:290] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:52.757786 28201 tablet_bootstrap.cc:654] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:52.759307 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:58:52.761415 27975 leader_election.cc:343] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:52.761826 27975 leader_election.cc:304] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:52.762076 28200 ts_tablet_manager.cc:1428] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.034s user 0.006s sys 0.000s
I20250905 19:58:52.762609 28393 raft_consensus.cc:2747] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:52.763471 28200 tablet_bootstrap.cc:492] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:52.769078 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:52.770159 28200 tablet_bootstrap.cc:654] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:52.770318 28008 raft_consensus.cc:2391] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 1: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:58:52.776387 28125 leader_election.cc:304] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:52.780246 28375 raft_consensus.cc:3058] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:52.787658 28375 raft_consensus.cc:2747] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:52.789508 28008 raft_consensus.cc:3058] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:52.790360 28200 tablet_bootstrap.cc:492] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:52.790913 28200 ts_tablet_manager.cc:1397] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.005s
I20250905 19:58:52.793885 28200 raft_consensus.cc:357] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.794755 28200 raft_consensus.cc:383] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:52.795100 28200 raft_consensus.cc:738] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:52.795884 28200 consensus_queue.cc:260] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.798867 28200 ts_tablet_manager.cc:1428] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250905 19:58:52.800079 28200 tablet_bootstrap.cc:492] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:52.800747 28370 consensus_queue.cc:1035] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:52.802676 28375 consensus_queue.cc:1035] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:52.813568 28200 tablet_bootstrap.cc:654] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:52.825173 28370 consensus_queue.cc:1035] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:52.827217 28200 tablet_bootstrap.cc:492] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:52.827767 28200 ts_tablet_manager.cc:1397] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.000s
I20250905 19:58:52.830024 28436 consensus_queue.cc:1035] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:52.836197 28200 raft_consensus.cc:357] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.836968 28200 raft_consensus.cc:383] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:52.837244 28200 raft_consensus.cc:738] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:52.843032 28200 consensus_queue.cc:260] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.848165 28200 ts_tablet_manager.cc:1428] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.020s user 0.005s sys 0.003s
I20250905 19:58:52.849116 28200 tablet_bootstrap.cc:492] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:52.849645 28370 raft_consensus.cc:491] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:52.850137 28370 raft_consensus.cc:513] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.852044 28370 leader_election.cc:290] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:52.853196 28373 consensus_queue.cc:1035] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:52.853638 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:52.858291 28200 tablet_bootstrap.cc:654] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:52.858320 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:52.861855 28051 leader_election.cc:304] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:52.864197 28370 raft_consensus.cc:2747] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:52.872038 28200 tablet_bootstrap.cc:492] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:52.872678 28200 ts_tablet_manager.cc:1397] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.000s
I20250905 19:58:52.874322 28369 consensus_queue.cc:1035] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:52.875775 28200 raft_consensus.cc:357] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.876727 28200 raft_consensus.cc:383] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:52.877075 28200 raft_consensus.cc:738] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:52.877803 28200 consensus_queue.cc:260] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.887709 28200 ts_tablet_manager.cc:1428] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20250905 19:58:52.907992 28202 tablet_bootstrap.cc:492] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:52.908424 28202 ts_tablet_manager.cc:1397] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.168s user 0.037s sys 0.035s
I20250905 19:58:52.908315 28370 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:52.910018 28200 tablet_bootstrap.cc:492] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:52.908949 28370 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.913973 28370 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:52.915035 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:52.915700 28159 raft_consensus.cc:2466] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:52.917253 28202 raft_consensus.cc:357] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.917944 28202 raft_consensus.cc:383] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:52.918210 28202 raft_consensus.cc:738] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:52.918833 28202 consensus_queue.cc:260] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.919564 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:58:52.920926 28051 leader_election.cc:343] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:52.924897 28200 tablet_bootstrap.cc:654] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:52.927083 28050 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:52.927877 28369 raft_consensus.cc:2802] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:52.928251 28369 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:52.928632 28201 tablet_bootstrap.cc:492] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:52.928606 28369 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:52.929206 28201 ts_tablet_manager.cc:1397] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.178s user 0.045s sys 0.045s
I20250905 19:58:52.931538 28201 raft_consensus.cc:357] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.932132 28201 raft_consensus.cc:383] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:52.932335 28201 raft_consensus.cc:738] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:52.932840 28201 consensus_queue.cc:260] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.936935 28369 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.940152 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:52.941068 28202 ts_tablet_manager.cc:1428] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.032s user 0.001s sys 0.005s
W20250905 19:58:52.941409 28051 leader_election.cc:343] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:52.941725 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:52.942185 28202 tablet_bootstrap.cc:492] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:52.942287 28159 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:52.950291 28369 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:52.953540 28202 tablet_bootstrap.cc:654] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:52.961208 28369 consensus_queue.cc:1035] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:52.962606 28159 raft_consensus.cc:2466] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:52.964946 28050 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:52.965420 28201 ts_tablet_manager.cc:1428] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.036s user 0.004s sys 0.000s
I20250905 19:58:52.966751 28369 raft_consensus.cc:2802] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:52.967288 28369 raft_consensus.cc:695] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:58:52.967444 28201 tablet_bootstrap.cc:492] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:52.967841 28200 tablet_bootstrap.cc:492] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:52.968338 28200 ts_tablet_manager.cc:1397] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.059s user 0.033s sys 0.007s
I20250905 19:58:52.968150 28369 consensus_queue.cc:237] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.970965 28200 raft_consensus.cc:357] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.971684 28200 raft_consensus.cc:383] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:52.971979 28200 raft_consensus.cc:738] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:52.973120 28200 consensus_queue.cc:260] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.974740 28201 tablet_bootstrap.cc:654] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:52.977394 28370 raft_consensus.cc:491] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:52.978086 28370 raft_consensus.cc:513] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.979502 28200 ts_tablet_manager.cc:1428] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.011s user 0.004s sys 0.004s
I20250905 19:58:52.980268 28370 leader_election.cc:290] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:52.980556 28200 tablet_bootstrap.cc:492] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:52.982213 28393 raft_consensus.cc:491] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:52.982723 28393 raft_consensus.cc:513] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:52.984994 28393 leader_election.cc:290] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:52.985776 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:52.986454 28083 raft_consensus.cc:2466] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 0.
I20250905 19:58:52.987183 28201 tablet_bootstrap.cc:492] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:52.987654 27973 leader_election.cc:304] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:52.988956 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:52.988883 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:52.989805 28157 raft_consensus.cc:2466] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 0.
I20250905 19:58:52.988881 28393 raft_consensus.cc:2802] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:52.990602 28006 raft_consensus.cc:2466] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:52.991488 28393 raft_consensus.cc:491] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:52.992013 28393 raft_consensus.cc:3058] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:52.995805 28200 tablet_bootstrap.cc:654] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:53.004945 28200 tablet_bootstrap.cc:492] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:53.005443 28200 ts_tablet_manager.cc:1397] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.003s
I20250905 19:58:53.005358 28051 leader_election.cc:304] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:53.006798 28436 raft_consensus.cc:2802] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:53.007206 28436 raft_consensus.cc:491] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:53.007546 28436 raft_consensus.cc:3058] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:53.008280 28200 raft_consensus.cc:357] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.009027 28200 raft_consensus.cc:383] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:53.009296 28200 raft_consensus.cc:738] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:53.014873 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:53.015694 28159 raft_consensus.cc:2466] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:52.987706 28201 ts_tablet_manager.cc:1397] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.008s
I20250905 19:58:53.009967 28200 consensus_queue.cc:260] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.019995 28200 ts_tablet_manager.cc:1428] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20250905 19:58:53.019797 28201 raft_consensus.cc:357] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.020603 28201 raft_consensus.cc:383] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:53.020864 28201 raft_consensus.cc:738] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:53.021003 28200 tablet_bootstrap.cc:492] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:53.021451 28201 consensus_queue.cc:260] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.024706 28201 ts_tablet_manager.cc:1428] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250905 19:58:53.026212 28201 tablet_bootstrap.cc:492] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:53.027755 27905 catalog_manager.cc:5582] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:53.032361 28375 raft_consensus.cc:491] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:53.033074 28375 raft_consensus.cc:513] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.036628 28375 leader_election.cc:290] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:53.033597 28200 tablet_bootstrap.cc:654] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:53.037933 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:53.038750 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:53.038825 28436 raft_consensus.cc:513] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:58:53.040117 28125 leader_election.cc:343] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:53.037967 28370 consensus_queue.cc:1035] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
W20250905 19:58:53.041030 28122 leader_election.cc:343] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:53.041472 28122 leader_election.cc:304] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:53.043181 28375 raft_consensus.cc:2747] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:53.053709 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:53.054510 28157 raft_consensus.cc:3058] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:53.056948 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:53.057840 28201 tablet_bootstrap.cc:654] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:53.060941 28200 tablet_bootstrap.cc:492] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:53.063793 28393 raft_consensus.cc:513] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.064889 28157 raft_consensus.cc:2466] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:53.065160 28008 raft_consensus.cc:2391] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 1: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:58:53.066092 28050 leader_election.cc:304] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:53.067003 28370 raft_consensus.cc:2802] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:53.067468 28370 raft_consensus.cc:695] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:58:53.061390 28200 ts_tablet_manager.cc:1397] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.041s user 0.010s sys 0.000s
I20250905 19:58:53.072532 28370 consensus_queue.cc:237] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.073765 28200 raft_consensus.cc:357] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.074566 28200 raft_consensus.cc:383] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:53.074853 28200 raft_consensus.cc:738] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:53.077008 28202 tablet_bootstrap.cc:492] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:53.077410 28202 ts_tablet_manager.cc:1397] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.135s user 0.033s sys 0.019s
I20250905 19:58:53.075493 28200 consensus_queue.cc:260] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.080140 28393 leader_election.cc:290] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:53.083014 28201 tablet_bootstrap.cc:492] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:53.083175 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:58:53.083772 28201 ts_tablet_manager.cc:1397] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.058s user 0.009s sys 0.003s
I20250905 19:58:53.086057 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:53.090499 28159 raft_consensus.cc:2391] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 1: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:58:53.086359 28202 raft_consensus.cc:357] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.076506 28436 leader_election.cc:290] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:53.091538 28202 raft_consensus.cc:383] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:53.091583 27975 leader_election.cc:304] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:53.091949 28202 raft_consensus.cc:738] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:53.092836 28404 raft_consensus.cc:2747] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250905 19:58:53.093457 28404 raft_consensus.cc:491] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:53.093291 28202 consensus_queue.cc:260] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.093740 28375 raft_consensus.cc:491] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:53.093926 28404 raft_consensus.cc:513] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.094221 28375 raft_consensus.cc:513] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.089555 28200 ts_tablet_manager.cc:1428] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.021s user 0.006s sys 0.001s
I20250905 19:58:53.096087 28404 leader_election.cc:290] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:53.096499 28375 leader_election.cc:290] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:53.097046 28202 ts_tablet_manager.cc:1428] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.019s user 0.005s sys 0.001s
I20250905 19:58:53.097998 28202 tablet_bootstrap.cc:492] T 8e0ac2dfe59d423189dd8d72eee98a42 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:53.097829 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:53.093044 28201 raft_consensus.cc:357] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.098412 28201 raft_consensus.cc:383] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:53.099013 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:58:53.099714 27973 leader_election.cc:343] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:53.098786 28201 raft_consensus.cc:738] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:53.100488 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:58:53.100739 28125 leader_election.cc:343] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:53.100790 28201 consensus_queue.cc:260] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:58:53.101831 28122 leader_election.cc:343] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:53.102298 28122 leader_election.cc:304] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:53.103480 28200 tablet_bootstrap.cc:492] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:53.103633 28375 raft_consensus.cc:2747] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:53.104766 28201 ts_tablet_manager.cc:1428] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.015s user 0.003s sys 0.003s
I20250905 19:58:53.106163 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:53.106838 28157 raft_consensus.cc:2466] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 0.
I20250905 19:58:53.107524 28201 tablet_bootstrap.cc:492] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:53.111711 27975 leader_election.cc:304] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:53.113111 28202 tablet_bootstrap.cc:654] T 8e0ac2dfe59d423189dd8d72eee98a42 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:53.113287 28404 raft_consensus.cc:2802] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:53.113638 28404 raft_consensus.cc:491] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:53.114221 28404 raft_consensus.cc:3058] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:53.127218 28404 raft_consensus.cc:513] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.132658 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 19:58:53.134522 27973 leader_election.cc:343] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:53.134053 27906 catalog_manager.cc:5582] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:53.137400 28404 leader_election.cc:290] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:53.138602 28201 tablet_bootstrap.cc:654] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:53.140331 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:53.141732 28159 raft_consensus.cc:3058] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:53.150166 28200 tablet_bootstrap.cc:654] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:53.154824 28375 raft_consensus.cc:491] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:53.155606 28375 raft_consensus.cc:513] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.159230 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:53.159577 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:53.160346 28081 raft_consensus.cc:2391] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 1: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:58:53.161844 28122 leader_election.cc:304] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:53.163020 28373 raft_consensus.cc:3058] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:53.163143 28375 leader_election.cc:290] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:53.163769 28202 tablet_bootstrap.cc:492] T 8e0ac2dfe59d423189dd8d72eee98a42 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:53.164265 28202 ts_tablet_manager.cc:1397] T 8e0ac2dfe59d423189dd8d72eee98a42 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.066s user 0.014s sys 0.010s
I20250905 19:58:53.168699 28202 raft_consensus.cc:357] T 8e0ac2dfe59d423189dd8d72eee98a42 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.170506 28202 raft_consensus.cc:383] T 8e0ac2dfe59d423189dd8d72eee98a42 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:53.170758 28202 raft_consensus.cc:738] T 8e0ac2dfe59d423189dd8d72eee98a42 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:53.171532 28202 consensus_queue.cc:260] T 8e0ac2dfe59d423189dd8d72eee98a42 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.164501 28159 raft_consensus.cc:2466] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:58:53.175158 28202 ts_tablet_manager.cc:1428] T 8e0ac2dfe59d423189dd8d72eee98a42 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.011s user 0.004s sys 0.006s
I20250905 19:58:53.176288 28202 tablet_bootstrap.cc:492] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:53.176169 27975 leader_election.cc:304] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:53.177318 28393 raft_consensus.cc:2802] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:53.177837 28393 raft_consensus.cc:695] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:58:53.178686 28393 consensus_queue.cc:237] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.180145 28373 raft_consensus.cc:2747] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:53.196605 28200 tablet_bootstrap.cc:492] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:53.197120 28200 ts_tablet_manager.cc:1397] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.101s user 0.015s sys 0.023s
I20250905 19:58:53.193621 28201 tablet_bootstrap.cc:492] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:53.198807 28201 ts_tablet_manager.cc:1397] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.092s user 0.021s sys 0.016s
I20250905 19:58:53.199528 28200 raft_consensus.cc:357] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.200263 28200 raft_consensus.cc:383] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:53.200541 28200 raft_consensus.cc:738] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:53.195919 28202 tablet_bootstrap.cc:654] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:53.201275 28200 consensus_queue.cc:260] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.201776 28201 raft_consensus.cc:357] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.202505 28201 raft_consensus.cc:383] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:53.202778 28201 raft_consensus.cc:738] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:53.203372 28201 consensus_queue.cc:260] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.206660 27906 catalog_manager.cc:5582] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:53.209337 28373 raft_consensus.cc:491] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:53.209860 28373 raft_consensus.cc:513] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.212320 28373 leader_election.cc:290] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:53.217000 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:53.217715 28081 raft_consensus.cc:2466] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 0.
I20250905 19:58:53.219004 28200 ts_tablet_manager.cc:1428] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20250905 19:58:53.220105 28200 tablet_bootstrap.cc:492] T 5ed920d4c0144d62acf66f595bbf34fd P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:53.224503 28122 leader_election.cc:304] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:53.225982 28375 raft_consensus.cc:2802] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:53.226444 28375 raft_consensus.cc:491] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:53.226787 28375 raft_consensus.cc:3058] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:53.227934 28201 ts_tablet_manager.cc:1428] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20250905 19:58:53.228919 28201 tablet_bootstrap.cc:492] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:53.226119 28200 tablet_bootstrap.cc:654] T 5ed920d4c0144d62acf66f595bbf34fd P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:53.236229 28366 raft_consensus.cc:491] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:53.236913 28366 raft_consensus.cc:513] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.239161 28201 tablet_bootstrap.cc:654] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:53.239500 28376 raft_consensus.cc:491] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:53.240134 28300 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:53.240089 28376 raft_consensus.cc:513] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.240641 28300 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.241775 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:53.242084 28376 leader_election.cc:290] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:53.242538 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:53.242981 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:53.243254 28006 raft_consensus.cc:2466] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 0.
I20250905 19:58:53.243577 28008 raft_consensus.cc:2466] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 0.
I20250905 19:58:53.239553 28366 leader_election.cc:290] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:53.244071 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:53.244628 28159 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:58:53.246321 28202 tablet_bootstrap.cc:492] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:53.246527 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:53.246855 28202 ts_tablet_manager.cc:1397] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.071s user 0.028s sys 0.015s
I20250905 19:58:53.246891 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:53.248605 27975 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:58:53.250385 28404 raft_consensus.cc:2802] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250905 19:58:53.250679 28122 leader_election.cc:343] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:53.251056 28404 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:53.251555 28404 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:53.252099 28393 consensus_queue.cc:1035] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:53.256330 28200 tablet_bootstrap.cc:492] T 5ed920d4c0144d62acf66f595bbf34fd P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:53.256227 28125 leader_election.cc:304] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:53.247188 28157 raft_consensus.cc:2466] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 0.
I20250905 19:58:53.256873 28200 ts_tablet_manager.cc:1397] T 5ed920d4c0144d62acf66f595bbf34fd P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.004s
I20250905 19:58:53.242576 28300 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:53.258162 28366 raft_consensus.cc:2802] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:53.258589 28366 raft_consensus.cc:491] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:53.258948 28366 raft_consensus.cc:3058] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:53.259532 28369 raft_consensus.cc:491] T 8e0ac2dfe59d423189dd8d72eee98a42 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:53.260059 28369 raft_consensus.cc:513] T 8e0ac2dfe59d423189dd8d72eee98a42 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.260517 28081 raft_consensus.cc:3058] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:53.262825 27975 leader_election.cc:304] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 19:58:53.263535 27973 leader_election.cc:343] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:53.263777 28300 raft_consensus.cc:2802] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:53.264147 28300 raft_consensus.cc:491] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:53.264487 28300 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:53.269024 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:53.269502 28202 raft_consensus.cc:357] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.270197 28202 raft_consensus.cc:383] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:53.270534 28202 raft_consensus.cc:738] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:53.263022 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:53.271245 28202 consensus_queue.cc:260] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.274752 28368 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:58:53.259701 28200 raft_consensus.cc:357] T 5ed920d4c0144d62acf66f595bbf34fd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.275897 28200 raft_consensus.cc:383] T 5ed920d4c0144d62acf66f595bbf34fd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:53.276183 28200 raft_consensus.cc:738] T 5ed920d4c0144d62acf66f595bbf34fd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:53.275307 28368 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.276866 28200 consensus_queue.cc:260] T 5ed920d4c0144d62acf66f595bbf34fd P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.279013 28368 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:53.281195 28404 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.282560 28376 consensus_queue.cc:1035] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:53.284246 28404 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:53.284857 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:58:53.285673 28083 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 3
I20250905 19:58:53.279237 28200 ts_tablet_manager.cc:1428] T 5ed920d4c0144d62acf66f595bbf34fd P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20250905 19:58:53.291342 28370 raft_consensus.cc:491] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:53.295253 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:53.295447 28370 raft_consensus.cc:513] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.296270 28200 tablet_bootstrap.cc:492] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:53.296358 28369 leader_election.cc:290] T 8e0ac2dfe59d423189dd8d72eee98a42 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:53.297843 28370 leader_election.cc:290] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:53.294497 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:53.293255 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:53.295878 28157 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:58:53.305495 28158 raft_consensus.cc:2391] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 1: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:58:53.305815 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:53.307036 28008 raft_consensus.cc:2366] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 2. Current term is 3.
I20250905 19:58:53.309711 28202 ts_tablet_manager.cc:1428] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.062s user 0.002s sys 0.004s
I20250905 19:58:53.325742 28202 tablet_bootstrap.cc:492] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:53.326210 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:53.326902 28159 raft_consensus.cc:2466] T 5ed920d4c0144d62acf66f595bbf34fd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:53.325621 28050 leader_election.cc:304] T 8e0ac2dfe59d423189dd8d72eee98a42 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:53.336585 28083 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:58:53.338119 27973 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:53.333323 28369 raft_consensus.cc:3058] T 8e0ac2dfe59d423189dd8d72eee98a42 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:53.339243 28376 raft_consensus.cc:2802] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:58:53.340010 28376 raft_consensus.cc:695] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:58:53.341130 28376 consensus_queue.cc:237] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.324658 28375 raft_consensus.cc:513] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.344144 28375 leader_election.cc:290] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:53.322059 28366 raft_consensus.cc:513] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.345289 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:53.346127 28006 raft_consensus.cc:3058] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:53.346522 28366 leader_election.cc:290] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:53.354631 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:53.354780 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:53.356326 28008 raft_consensus.cc:2466] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:53.356562 28007 raft_consensus.cc:3058] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:53.358088 28050 leader_election.cc:304] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:53.359627 28370 raft_consensus.cc:2802] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:53.359989 28370 raft_consensus.cc:491] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:53.360296 28370 raft_consensus.cc:3058] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:53.365269 28369 raft_consensus.cc:2747] T 8e0ac2dfe59d423189dd8d72eee98a42 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:53.350773 28300 raft_consensus.cc:513] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.369585 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:58:53.453238 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:53.455753 28159 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:53.462383 28201 tablet_bootstrap.cc:492] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:53.462957 28201 ts_tablet_manager.cc:1397] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.234s user 0.019s sys 0.011s
I20250905 19:58:53.465489 28157 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:58:53.432675 28366 raft_consensus.cc:491] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:53.469937 28366 raft_consensus.cc:513] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.471694 28201 raft_consensus.cc:357] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.472442 28201 raft_consensus.cc:383] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:53.472756 28201 raft_consensus.cc:738] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:53.472538 28366 leader_election.cc:290] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:53.378883 28393 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:53.407655 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:58:53.373106 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 19:58:53.475821 27973 leader_election.cc:343] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:53.466873 28007 raft_consensus.cc:2466] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 19:58:53.374519 28300 leader_election.cc:290] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:53.465323 28083 raft_consensus.cc:3058] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:53.480788 28200 tablet_bootstrap.cc:654] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:53.474154 28393 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.483219 28201 consensus_queue.cc:260] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.465952 28006 raft_consensus.cc:2466] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 19:58:53.488108 28159 raft_consensus.cc:2466] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:58:53.491040 28202 tablet_bootstrap.cc:654] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:53.480252 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:53.479854 28125 leader_election.cc:304] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:58:53.497444 28201 ts_tablet_manager.cc:1428] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.034s user 0.007s sys 0.001s
I20250905 19:58:53.498523 28201 tablet_bootstrap.cc:492] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:53.493655 28370 raft_consensus.cc:513] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.488718 28368 consensus_queue.cc:1035] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20250905 19:58:53.502319 28125 leader_election.cc:304] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 19:58:53.505213 28122 leader_election.cc:343] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:53.509333 28393 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:58:53.509334 28125 leader_election.cc:343] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:53.510139 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:53.512692 28373 raft_consensus.cc:2802] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:53.506134 27975 leader_election.cc:304] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:53.515539 28051 leader_election.cc:400] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Vote denied by peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421) with higher term. Message: Invalid argument: T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 2. Current term is 3.
I20250905 19:58:53.515928 28051 leader_election.cc:403] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250905 19:58:53.517562 28436 raft_consensus.cc:2747] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421) with higher term. Message: Invalid argument: T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 2. Current term is 3.
I20250905 19:58:53.519795 28370 leader_election.cc:290] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:53.523231 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:53.524541 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:53.525183 28157 raft_consensus.cc:3058] T 5ed920d4c0144d62acf66f595bbf34fd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:53.501746 28366 raft_consensus.cc:2802] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:53.528332 28366 raft_consensus.cc:695] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:58:53.534224 28006 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:53.532300 28487 raft_consensus.cc:2802] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:53.537801 28366 consensus_queue.cc:237] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.541919 28487 raft_consensus.cc:695] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:58:53.545315 28200 tablet_bootstrap.cc:492] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:53.545826 28200 ts_tablet_manager.cc:1397] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.250s user 0.015s sys 0.006s
I20250905 19:58:53.557224 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:53.560767 27907 catalog_manager.cc:5582] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:53.552608 28373 raft_consensus.cc:695] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:58:53.560297 28083 raft_consensus.cc:2466] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 19:58:53.577028 28157 raft_consensus.cc:2466] T 5ed920d4c0144d62acf66f595bbf34fd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:53.577317 28201 tablet_bootstrap.cc:654] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:53.578814 28200 raft_consensus.cc:357] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.579368 28200 raft_consensus.cc:383] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:53.579562 28200 raft_consensus.cc:738] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:53.580125 28200 consensus_queue.cc:260] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.581753 28202 tablet_bootstrap.cc:492] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:53.582152 28202 ts_tablet_manager.cc:1397] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.257s user 0.014s sys 0.005s
I20250905 19:58:53.542851 28487 consensus_queue.cc:237] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.586249 28373 consensus_queue.cc:237] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:58:53.577764 27956 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250905 19:58:53.587929 28200 ts_tablet_manager.cc:1428] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.042s user 0.006s sys 0.001s
I20250905 19:58:53.591112 28200 tablet_bootstrap.cc:492] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:53.593290 27975 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:53.603880 28050 leader_election.cc:304] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:53.592139 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:53.601814 28393 raft_consensus.cc:2747] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:58:53.602737 28122 leader_election.cc:343] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:53.613583 28202 raft_consensus.cc:357] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.616495 28202 raft_consensus.cc:383] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:53.616835 28202 raft_consensus.cc:738] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:53.617522 28202 consensus_queue.cc:260] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.611276 28005 raft_consensus.cc:3058] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:53.610458 28376 raft_consensus.cc:491] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:53.632503 28202 ts_tablet_manager.cc:1428] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.050s user 0.007s sys 0.000s
I20250905 19:58:53.619424 28435 raft_consensus.cc:2802] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:53.633792 28202 tablet_bootstrap.cc:492] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:53.621942 28122 leader_election.cc:304] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:53.632462 28376 raft_consensus.cc:513] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.658483 28375 raft_consensus.cc:2747] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:58:53.663053 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 64955.5us
I20250905 19:58:53.680022 28376 leader_election.cc:290] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:53.682137 28436 consensus_queue.cc:1035] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20250905 19:58:53.696511 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:53.703609 28436 consensus_queue.cc:1035] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:53.701836 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:53.678647 28494 consensus_queue.cc:1035] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20250905 19:58:53.736096 28200 tablet_bootstrap.cc:654] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:53.736877 28202 tablet_bootstrap.cc:654] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
W20250905 19:58:53.748627 27973 leader_election.cc:343] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:58:53.750957 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 85417.4us
W20250905 19:58:53.774685 27975 leader_election.cc:343] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:53.775198 27975 leader_election.cc:304] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:53.772511 28435 raft_consensus.cc:695] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:58:53.792073 28201 tablet_bootstrap.cc:492] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:53.793576 28201 ts_tablet_manager.cc:1397] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.295s user 0.009s sys 0.013s
I20250905 19:58:53.768568 28005 raft_consensus.cc:2466] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:53.795682 28376 consensus_queue.cc:1035] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250905 19:58:53.784296 28487 raft_consensus.cc:2747] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:53.762637 28504 consensus_queue.cc:1035] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20250905 19:58:53.820017 27907 catalog_manager.cc:5582] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:53.792286 28435 consensus_queue.cc:237] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.828141 28201 raft_consensus.cc:357] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.829779 28201 raft_consensus.cc:383] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:53.830183 28201 raft_consensus.cc:738] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:53.835160 28201 consensus_queue.cc:260] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.841552 28200 tablet_bootstrap.cc:492] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:53.842073 28200 ts_tablet_manager.cc:1397] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.251s user 0.014s sys 0.014s
I20250905 19:58:53.842096 28201 ts_tablet_manager.cc:1428] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.048s user 0.008s sys 0.000s
I20250905 19:58:53.854866 28201 tablet_bootstrap.cc:492] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:53.853983 27905 catalog_manager.cc:5582] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20250905 19:58:53.860015 27905 catalog_manager.cc:5582] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20250905 19:58:53.863057 28200 raft_consensus.cc:357] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.865265 28200 raft_consensus.cc:383] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:53.866199 28200 raft_consensus.cc:738] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:53.869521 28200 consensus_queue.cc:260] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.878163 28202 tablet_bootstrap.cc:492] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:53.878634 28202 ts_tablet_manager.cc:1397] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.245s user 0.022s sys 0.010s
I20250905 19:58:53.889007 28200 ts_tablet_manager.cc:1428] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.047s user 0.008s sys 0.000s
I20250905 19:58:53.890920 28202 raft_consensus.cc:357] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.891673 28202 raft_consensus.cc:383] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:53.891968 28202 raft_consensus.cc:738] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:53.892573 28202 consensus_queue.cc:260] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.894524 28201 tablet_bootstrap.cc:654] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:53.894934 28202 ts_tablet_manager.cc:1428] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20250905 19:58:53.896070 28202 tablet_bootstrap.cc:492] T fd2b679c53994263846f51ceec97a150 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:53.901249 28200 tablet_bootstrap.cc:492] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:53.904036 27907 catalog_manager.cc:5582] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:53.908164 28202 tablet_bootstrap.cc:654] T fd2b679c53994263846f51ceec97a150 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:53.910105 28366 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:53.910552 28366 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.914168 28200 tablet_bootstrap.cc:654] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:53.923892 28366 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:53.928544 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:53.930337 28157 raft_consensus.cc:1273] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250905 19:58:53.930586 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:58:53.931448 28122 leader_election.cc:343] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:53.932104 28005 raft_consensus.cc:2466] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 0.
I20250905 19:58:53.934736 28125 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:53.938627 28366 raft_consensus.cc:2802] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:53.939143 28366 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:53.939579 28366 raft_consensus.cc:3058] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:53.952369 28506 raft_consensus.cc:491] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:53.951478 28487 consensus_queue.cc:1035] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.011s
I20250905 19:58:53.974397 28202 tablet_bootstrap.cc:492] T fd2b679c53994263846f51ceec97a150 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:53.975533 28202 ts_tablet_manager.cc:1397] T fd2b679c53994263846f51ceec97a150 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.080s user 0.027s sys 0.020s
I20250905 19:58:53.963093 28506 raft_consensus.cc:513] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.979923 28506 leader_election.cc:290] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:53.985472 28200 tablet_bootstrap.cc:492] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:53.986129 28200 ts_tablet_manager.cc:1397] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.096s user 0.026s sys 0.005s
I20250905 19:58:53.988071 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:53.989061 28201 tablet_bootstrap.cc:492] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:53.989534 28201 ts_tablet_manager.cc:1397] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.135s user 0.014s sys 0.004s
I20250905 19:58:53.994838 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:53.995618 28159 raft_consensus.cc:2391] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 1: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:58:53.974046 28366 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.998543 28436 raft_consensus.cc:491] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:53.999223 28436 raft_consensus.cc:513] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:53.999986 28366 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:54.000283 28083 raft_consensus.cc:1273] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250905 19:58:54.000680 28201 raft_consensus.cc:357] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.001153 28436 leader_election.cc:290] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:54.001369 28201 raft_consensus.cc:383] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:54.001706 28201 raft_consensus.cc:738] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:54.002355 28201 consensus_queue.cc:260] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.003023 28050 leader_election.cc:304] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:54.004532 28201 ts_tablet_manager.cc:1428] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.015s user 0.004s sys 0.000s
I20250905 19:58:54.004789 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:54.005437 28201 tablet_bootstrap.cc:492] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:54.005760 28375 consensus_queue.cc:1035] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:54.005388 28005 raft_consensus.cc:3058] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:54.006126 28300 consensus_queue.cc:1035] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20250905 19:58:54.013935 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:58:54.014986 28202 raft_consensus.cc:357] T fd2b679c53994263846f51ceec97a150 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.015753 28202 raft_consensus.cc:383] T fd2b679c53994263846f51ceec97a150 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:54.016244 28202 raft_consensus.cc:738] T fd2b679c53994263846f51ceec97a150 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:54.017412 28202 consensus_queue.cc:260] T fd2b679c53994263846f51ceec97a150 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.020514 28501 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:58:54.022642 28122 leader_election.cc:343] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:54.023643 28200 raft_consensus.cc:357] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.024029 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:54.025003 28200 raft_consensus.cc:383] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:54.025306 28200 raft_consensus.cc:738] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:54.032785 28200 consensus_queue.cc:260] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.039688 28202 ts_tablet_manager.cc:1428] T fd2b679c53994263846f51ceec97a150 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.064s user 0.006s sys 0.000s
I20250905 19:58:54.040833 28202 tablet_bootstrap.cc:492] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:54.042382 28004 raft_consensus.cc:2391] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 1: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:58:54.048466 28366 raft_consensus.cc:491] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:54.062171 28366 raft_consensus.cc:513] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.065196 28366 leader_election.cc:290] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:54.065960 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:54.067939 28501 raft_consensus.cc:2747] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:58:54.068318 28125 leader_election.cc:343] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:54.070820 28200 ts_tablet_manager.cc:1428] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.084s user 0.003s sys 0.005s
I20250905 19:58:54.071646 28200 tablet_bootstrap.cc:492] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:54.075819 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:54.077157 28081 raft_consensus.cc:3058] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:58:54.077344 28122 leader_election.cc:343] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:54.077845 28122 leader_election.cc:304] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:54.070096 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:54.081720 28491 raft_consensus.cc:2747] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:54.101197 28005 raft_consensus.cc:2466] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 19:58:54.102497 28125 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:54.106483 28488 raft_consensus.cc:2802] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:54.107064 28488 raft_consensus.cc:695] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:58:54.110124 28050 leader_election.cc:304] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:54.110364 28488 consensus_queue.cc:237] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.111811 28436 raft_consensus.cc:2747] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:54.115947 28373 consensus_queue.cc:1035] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:54.117499 28373 consensus_queue.cc:1035] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:54.124679 28200 tablet_bootstrap.cc:654] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:54.126798 28201 tablet_bootstrap.cc:654] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:54.131176 28376 consensus_queue.cc:1035] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:54.131927 28202 tablet_bootstrap.cc:654] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:54.132903 28487 raft_consensus.cc:491] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:54.133613 28487 raft_consensus.cc:513] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.135831 28487 leader_election.cc:290] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:54.137503 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:54.148767 28158 raft_consensus.cc:2466] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 0.
I20250905 19:58:54.149986 27975 leader_election.cc:304] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:58:54.139302 28376 consensus_queue.cc:1035] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:54.143119 28201 tablet_bootstrap.cc:492] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:54.153376 28201 ts_tablet_manager.cc:1397] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.148s user 0.013s sys 0.000s
I20250905 19:58:54.156137 28201 raft_consensus.cc:357] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.156836 28201 raft_consensus.cc:383] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:54.157094 28201 raft_consensus.cc:738] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:54.157712 28201 consensus_queue.cc:260] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.160228 28201 ts_tablet_manager.cc:1428] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250905 19:58:54.146813 28200 tablet_bootstrap.cc:492] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:54.146931 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:54.151232 28485 raft_consensus.cc:2802] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:54.161295 28485 raft_consensus.cc:491] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:54.161720 28485 raft_consensus.cc:3058] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:54.162384 28201 tablet_bootstrap.cc:492] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:54.163128 28375 consensus_queue.cc:1035] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:54.170419 28201 tablet_bootstrap.cc:654] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:54.176548 28501 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:58:54.177384 27973 leader_election.cc:343] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:54.160936 28200 ts_tablet_manager.cc:1397] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.089s user 0.015s sys 0.001s
I20250905 19:58:54.177132 28501 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.180291 28200 raft_consensus.cc:357] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.180970 28200 raft_consensus.cc:383] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:54.181236 28200 raft_consensus.cc:738] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:54.181915 28501 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:54.181859 28200 consensus_queue.cc:260] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.183859 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:54.184561 28201 tablet_bootstrap.cc:492] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:54.184741 28200 ts_tablet_manager.cc:1428] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250905 19:58:54.185297 28201 ts_tablet_manager.cc:1397] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.001s
I20250905 19:58:54.145610 28202 tablet_bootstrap.cc:492] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:54.185986 28202 ts_tablet_manager.cc:1397] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.145s user 0.008s sys 0.006s
I20250905 19:58:54.186831 28200 tablet_bootstrap.cc:492] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:54.189428 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:54.184654 28004 raft_consensus.cc:2391] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 1: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:58:54.193800 28201 raft_consensus.cc:357] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.195098 28201 raft_consensus.cc:383] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:54.195482 28201 raft_consensus.cc:738] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:54.196302 28201 consensus_queue.cc:260] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.199067 28485 raft_consensus.cc:513] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.201244 28485 leader_election.cc:290] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:54.202549 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:54.203434 28157 raft_consensus.cc:3058] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:54.205739 28050 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:54.194847 28202 raft_consensus.cc:357] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.207602 28202 raft_consensus.cc:383] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:54.207908 28202 raft_consensus.cc:738] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:54.208555 28436 raft_consensus.cc:3058] T fd2b679c53994263846f51ceec97a150 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:54.216324 28201 ts_tablet_manager.cc:1428] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.031s user 0.008s sys 0.001s
I20250905 19:58:54.217803 28201 tablet_bootstrap.cc:492] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:54.219229 28436 raft_consensus.cc:2747] T fd2b679c53994263846f51ceec97a150 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:54.221694 28157 raft_consensus.cc:2466] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:58:54.223106 27907 catalog_manager.cc:5582] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20250905 19:58:54.224998 28201 tablet_bootstrap.cc:654] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:54.225179 28200 tablet_bootstrap.cc:654] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:54.225396 27975 leader_election.cc:304] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:58:54.226159 28487 raft_consensus.cc:2802] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:54.226642 28487 raft_consensus.cc:695] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:58:54.227489 28487 consensus_queue.cc:237] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.221683 28202 consensus_queue.cc:260] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.240535 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 19:58:54.244896 27973 leader_election.cc:343] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:54.260169 28202 ts_tablet_manager.cc:1428] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.074s user 0.004s sys 0.001s
I20250905 19:58:54.266171 28202 tablet_bootstrap.cc:492] T 10df9d019f2e4e208f4e23d50f551ad6 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:54.272120 28435 consensus_queue.cc:1035] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:54.276588 28202 tablet_bootstrap.cc:654] T 10df9d019f2e4e208f4e23d50f551ad6 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:54.281760 28501 consensus_queue.cc:1035] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:54.292618 28201 tablet_bootstrap.cc:492] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:54.293248 28201 ts_tablet_manager.cc:1397] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.076s user 0.022s sys 0.034s
I20250905 19:58:54.277727 28200 tablet_bootstrap.cc:492] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:54.298518 28200 ts_tablet_manager.cc:1397] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.112s user 0.021s sys 0.003s
I20250905 19:58:54.304075 28200 raft_consensus.cc:357] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.296254 28201 raft_consensus.cc:357] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.308739 28201 raft_consensus.cc:383] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:54.309145 28201 raft_consensus.cc:738] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:54.309973 28201 consensus_queue.cc:260] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.312810 28201 ts_tablet_manager.cc:1428] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20250905 19:58:54.312891 28200 raft_consensus.cc:383] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:54.313429 28200 raft_consensus.cc:738] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:54.314019 28201 tablet_bootstrap.cc:492] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:54.315867 28200 consensus_queue.cc:260] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.320848 28201 tablet_bootstrap.cc:654] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:54.327817 28202 tablet_bootstrap.cc:492] T 10df9d019f2e4e208f4e23d50f551ad6 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:54.328387 28202 ts_tablet_manager.cc:1397] T 10df9d019f2e4e208f4e23d50f551ad6 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.062s user 0.011s sys 0.018s
I20250905 19:58:54.331292 28202 raft_consensus.cc:357] T 10df9d019f2e4e208f4e23d50f551ad6 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.332012 28202 raft_consensus.cc:383] T 10df9d019f2e4e208f4e23d50f551ad6 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:54.332278 28202 raft_consensus.cc:738] T 10df9d019f2e4e208f4e23d50f551ad6 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:54.336052 28375 raft_consensus.cc:491] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:54.336586 28375 raft_consensus.cc:513] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.338693 28375 leader_election.cc:290] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:54.339197 28375 raft_consensus.cc:491] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:54.339461 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:54.339641 28375 raft_consensus.cc:513] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.339993 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:58:54.340855 28125 leader_election.cc:343] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:54.341509 28375 leader_election.cc:290] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 19:58:54.341733 28122 leader_election.cc:343] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:54.342247 28122 leader_election.cc:304] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:54.342949 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:54.343576 28005 raft_consensus.cc:2466] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 0.
I20250905 19:58:54.344255 28491 raft_consensus.cc:2747] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:54.343626 27906 catalog_manager.cc:5582] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:54.344803 28125 leader_election.cc:304] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:58:54.345613 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:54.348294 28491 raft_consensus.cc:2802] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:54.348686 28491 raft_consensus.cc:491] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:54.349030 28491 raft_consensus.cc:3058] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:54.332975 28202 consensus_queue.cc:260] T 10df9d019f2e4e208f4e23d50f551ad6 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.350796 28080 raft_consensus.cc:2466] T 10df9d019f2e4e208f4e23d50f551ad6 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 0.
I20250905 19:58:54.355440 28202 ts_tablet_manager.cc:1428] T 10df9d019f2e4e208f4e23d50f551ad6 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.027s user 0.008s sys 0.000s
I20250905 19:58:54.356655 28202 tablet_bootstrap.cc:492] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:54.358238 28201 tablet_bootstrap.cc:492] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:54.358736 28201 ts_tablet_manager.cc:1397] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.045s user 0.012s sys 0.000s
I20250905 19:58:54.362205 28201 raft_consensus.cc:357] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.362907 28201 raft_consensus.cc:383] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:54.363176 28201 raft_consensus.cc:738] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:54.363852 28201 consensus_queue.cc:260] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.366469 28491 raft_consensus.cc:513] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.368526 28202 tablet_bootstrap.cc:654] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:54.372056 28487 raft_consensus.cc:491] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:54.372324 28491 leader_election.cc:290] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:54.372586 28487 raft_consensus.cc:513] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.373332 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:58:54.373931 28080 raft_consensus.cc:3058] T 10df9d019f2e4e208f4e23d50f551ad6 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:54.375444 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:54.376663 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 19:58:54.377139 27973 leader_election.cc:343] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:54.377496 28005 raft_consensus.cc:3058] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:54.374830 28487 leader_election.cc:290] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:54.381680 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:54.381954 28080 raft_consensus.cc:2466] T 10df9d019f2e4e208f4e23d50f551ad6 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 19:58:54.383424 28202 tablet_bootstrap.cc:492] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:54.383757 28157 raft_consensus.cc:2466] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 0.
I20250905 19:58:54.383904 28202 ts_tablet_manager.cc:1397] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.005s
I20250905 19:58:54.383949 28122 leader_election.cc:304] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:54.385044 27975 leader_election.cc:304] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:54.385733 28201 ts_tablet_manager.cc:1428] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20250905 19:58:54.386106 28487 raft_consensus.cc:2802] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:54.386521 28487 raft_consensus.cc:491] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:54.386795 28487 raft_consensus.cc:3058] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:54.387200 28202 raft_consensus.cc:357] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.387879 28202 raft_consensus.cc:383] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:54.388150 28202 raft_consensus.cc:738] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:54.389314 28375 raft_consensus.cc:2802] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:54.389868 28375 raft_consensus.cc:695] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:58:54.389969 28366 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:54.390525 28366 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.390743 28375 consensus_queue.cc:237] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.392516 28366 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:54.392973 28487 raft_consensus.cc:513] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.399983 28485 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:54.388767 28202 consensus_queue.cc:260] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.394670 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:54.400734 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:54.396551 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:58:54.401409 28006 raft_consensus.cc:2466] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 0.
I20250905 19:58:54.400627 28485 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.402614 28125 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 19:58:54.403224 28122 leader_election.cc:343] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:54.403508 28366 raft_consensus.cc:2802] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:54.403645 28485 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:54.403914 28366 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:54.404174 28201 tablet_bootstrap.cc:492] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:54.404328 28366 raft_consensus.cc:3058] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:54.405616 28200 ts_tablet_manager.cc:1428] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.104s user 0.004s sys 0.003s
I20250905 19:58:54.406638 28200 tablet_bootstrap.cc:492] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:54.396000 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 19:58:54.407130 27973 leader_election.cc:343] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:54.406702 28202 ts_tablet_manager.cc:1428] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.022s user 0.004s sys 0.002s
I20250905 19:58:54.411336 28202 tablet_bootstrap.cc:492] T c0889204295e4daa99d81e5b2a45ac1c P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:54.389249 28005 raft_consensus.cc:2466] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 19:58:54.409924 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:54.407564 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:54.407392 28157 raft_consensus.cc:3058] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:58:54.415028 27975 leader_election.cc:343] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:58:54.419443 27973 leader_election.cc:343] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:54.420020 27973 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:54.420923 28485 raft_consensus.cc:2747] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:54.422047 28202 tablet_bootstrap.cc:654] T c0889204295e4daa99d81e5b2a45ac1c P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:54.422804 28366 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.423995 28200 tablet_bootstrap.cc:654] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:54.424207 28487 leader_election.cc:290] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:54.428594 28157 raft_consensus.cc:2466] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:58:54.429742 27975 leader_election.cc:304] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:54.430879 28487 raft_consensus.cc:2802] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:54.431573 28487 raft_consensus.cc:695] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:58:54.432399 28487 consensus_queue.cc:237] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.437744 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:54.438118 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:58:54.438550 28005 raft_consensus.cc:3058] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:54.439385 28366 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 19:58:54.439585 28122 leader_election.cc:343] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:54.445489 27906 catalog_manager.cc:5582] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20250905 19:58:54.455142 28201 tablet_bootstrap.cc:654] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:54.456058 28202 tablet_bootstrap.cc:492] T c0889204295e4daa99d81e5b2a45ac1c P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:54.457130 28202 ts_tablet_manager.cc:1397] T c0889204295e4daa99d81e5b2a45ac1c P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.046s user 0.009s sys 0.012s
I20250905 19:58:54.459842 28202 raft_consensus.cc:357] T c0889204295e4daa99d81e5b2a45ac1c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.460531 28202 raft_consensus.cc:383] T c0889204295e4daa99d81e5b2a45ac1c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:54.460806 28202 raft_consensus.cc:738] T c0889204295e4daa99d81e5b2a45ac1c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:54.461401 28005 raft_consensus.cc:2466] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 19:58:54.461540 28202 consensus_queue.cc:260] T c0889204295e4daa99d81e5b2a45ac1c P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.462478 28125 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:54.463131 27907 catalog_manager.cc:5582] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:54.463840 28366 raft_consensus.cc:2802] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:54.464359 28366 raft_consensus.cc:695] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:58:54.465178 28366 consensus_queue.cc:237] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.472261 28202 ts_tablet_manager.cc:1428] T c0889204295e4daa99d81e5b2a45ac1c P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20250905 19:58:54.474136 28202 tablet_bootstrap.cc:492] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:54.477584 28201 tablet_bootstrap.cc:492] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:54.478193 28200 tablet_bootstrap.cc:492] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:54.478389 28201 ts_tablet_manager.cc:1397] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.076s user 0.008s sys 0.004s
I20250905 19:58:54.478706 28200 ts_tablet_manager.cc:1397] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.072s user 0.012s sys 0.001s
I20250905 19:58:54.482836 28201 raft_consensus.cc:357] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.483525 28201 raft_consensus.cc:383] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:54.483781 28201 raft_consensus.cc:738] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:54.484380 28201 consensus_queue.cc:260] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.486109 28200 raft_consensus.cc:357] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.486824 28200 raft_consensus.cc:383] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:54.487094 28200 raft_consensus.cc:738] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:54.486539 28201 ts_tablet_manager.cc:1428] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250905 19:58:54.487696 28202 tablet_bootstrap.cc:654] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:54.488415 28201 tablet_bootstrap.cc:492] T 8a8d8393352b4214a20c1683852d4fdd P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:54.489333 28200 consensus_queue.cc:260] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.500739 28201 tablet_bootstrap.cc:654] T 8a8d8393352b4214a20c1683852d4fdd P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:54.502413 27907 catalog_manager.cc:5582] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20250905 19:58:54.503441 28202 tablet_bootstrap.cc:492] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:54.510211 28200 ts_tablet_manager.cc:1428] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.027s user 0.005s sys 0.000s
I20250905 19:58:54.510599 28202 ts_tablet_manager.cc:1397] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.037s user 0.015s sys 0.006s
I20250905 19:58:54.511150 28504 raft_consensus.cc:491] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:54.511652 28504 raft_consensus.cc:513] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.511176 28200 tablet_bootstrap.cc:492] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:54.513652 28504 leader_election.cc:290] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:54.513841 28202 raft_consensus.cc:357] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.514564 28202 raft_consensus.cc:383] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:54.514829 28202 raft_consensus.cc:738] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:54.515403 28202 consensus_queue.cc:260] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.517676 28202 ts_tablet_manager.cc:1428] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.007s user 0.003s sys 0.001s
I20250905 19:58:54.519382 28202 tablet_bootstrap.cc:492] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:54.519508 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:54.520252 28157 raft_consensus.cc:2391] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 1: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:58:54.520267 28200 tablet_bootstrap.cc:654] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:54.526412 28202 tablet_bootstrap.cc:654] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:54.526535 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:54.529194 28051 leader_election.cc:304] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:54.530809 28436 raft_consensus.cc:3058] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:54.532567 28202 tablet_bootstrap.cc:492] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:54.533046 28202 ts_tablet_manager.cc:1397] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.001s
I20250905 19:58:54.533864 28201 tablet_bootstrap.cc:492] T 8a8d8393352b4214a20c1683852d4fdd P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:54.534324 28201 ts_tablet_manager.cc:1397] T 8a8d8393352b4214a20c1683852d4fdd P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.046s user 0.014s sys 0.000s
I20250905 19:58:54.535980 28202 raft_consensus.cc:357] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.536633 28202 raft_consensus.cc:383] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:54.536881 28202 raft_consensus.cc:738] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:54.537475 28202 consensus_queue.cc:260] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.540575 28201 raft_consensus.cc:357] T 8a8d8393352b4214a20c1683852d4fdd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.541345 28201 raft_consensus.cc:383] T 8a8d8393352b4214a20c1683852d4fdd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:54.541607 28201 raft_consensus.cc:738] T 8a8d8393352b4214a20c1683852d4fdd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:54.543520 28504 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:54.544061 28202 ts_tablet_manager.cc:1428] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.011s user 0.007s sys 0.002s
I20250905 19:58:54.544377 28436 raft_consensus.cc:2747] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:54.544021 28504 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.545470 28202 tablet_bootstrap.cc:492] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:54.542192 28201 consensus_queue.cc:260] T 8a8d8393352b4214a20c1683852d4fdd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.551404 28202 tablet_bootstrap.cc:654] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:54.553797 28201 ts_tablet_manager.cc:1428] T 8a8d8393352b4214a20c1683852d4fdd P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20250905 19:58:54.554996 28201 tablet_bootstrap.cc:492] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:54.559892 28366 consensus_queue.cc:1035] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:54.561192 28201 tablet_bootstrap.cc:654] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:54.565611 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:58:54.567273 28051 leader_election.cc:343] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:54.567363 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:54.568059 28157 raft_consensus.cc:2466] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:54.569172 28050 leader_election.cc:304] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:54.570562 28501 raft_consensus.cc:2802] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:54.570930 28501 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:54.571259 28501 raft_consensus.cc:3058] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:54.572610 28504 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:54.580117 28501 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.583091 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:54.584324 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 19:58:54.584509 28051 leader_election.cc:343] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:54.584920 28157 raft_consensus.cc:3058] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:54.590920 28202 tablet_bootstrap.cc:492] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:54.592303 28487 raft_consensus.cc:491] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:54.592830 28487 raft_consensus.cc:513] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.593513 28200 tablet_bootstrap.cc:492] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:54.593979 28200 ts_tablet_manager.cc:1397] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.083s user 0.019s sys 0.003s
I20250905 19:58:54.596004 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:54.596683 28158 raft_consensus.cc:2466] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 0.
I20250905 19:58:54.596608 28200 raft_consensus.cc:357] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.597352 28200 raft_consensus.cc:383] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:54.597608 28200 raft_consensus.cc:738] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:54.598224 28200 consensus_queue.cc:260] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.599893 28200 ts_tablet_manager.cc:1428] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20250905 19:58:54.600772 28200 tablet_bootstrap.cc:492] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:54.603065 28485 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:54.603582 28485 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.604622 28366 consensus_queue.cc:1035] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:54.605598 28485 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:54.607656 27975 leader_election.cc:304] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:58:54.607793 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:54.609138 28376 raft_consensus.cc:2802] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250905 19:58:54.609308 27975 leader_election.cc:343] T 8a8d8393352b4214a20c1683852d4fdd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:54.609555 28376 raft_consensus.cc:491] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:54.609936 28376 raft_consensus.cc:3058] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:54.610692 28487 leader_election.cc:290] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:54.614871 28200 tablet_bootstrap.cc:654] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:54.620412 28157 raft_consensus.cc:2466] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:54.622792 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:54.623575 28501 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:54.624197 28487 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:54.626116 28487 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.627918 28050 leader_election.cc:304] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:54.628152 28200 tablet_bootstrap.cc:492] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:54.628677 28200 ts_tablet_manager.cc:1397] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.028s user 0.012s sys 0.000s
I20250905 19:58:54.628983 28504 raft_consensus.cc:2802] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:54.629457 28504 raft_consensus.cc:695] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:58:54.629561 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:54.630260 28085 raft_consensus.cc:2466] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 0.
I20250905 19:58:54.630594 28436 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:54.631093 28436 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.631793 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:54.633263 28200 raft_consensus.cc:357] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:58:54.633602 27975 leader_election.cc:343] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:58:54.633703 27973 leader_election.cc:343] T 8a8d8393352b4214a20c1683852d4fdd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:54.636133 27973 leader_election.cc:304] T 8a8d8393352b4214a20c1683852d4fdd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:54.633978 28200 raft_consensus.cc:383] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:54.636741 28200 raft_consensus.cc:738] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:54.636803 28487 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:54.637481 28202 ts_tablet_manager.cc:1397] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.092s user 0.012s sys 0.007s
I20250905 19:58:54.637418 28200 consensus_queue.cc:260] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.639225 28200 ts_tablet_manager.cc:1428] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250905 19:58:54.640174 28202 raft_consensus.cc:357] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.640874 28202 raft_consensus.cc:383] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:54.641151 28202 raft_consensus.cc:738] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:54.633265 28436 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:54.642633 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:54.643621 28004 raft_consensus.cc:2391] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 1: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:58:54.643260 28504 consensus_queue.cc:237] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.640177 28200 tablet_bootstrap.cc:492] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:54.645571 28487 raft_consensus.cc:2747] T 8a8d8393352b4214a20c1683852d4fdd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:54.641800 28202 consensus_queue.cc:260] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.651592 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:54.652019 28376 raft_consensus.cc:513] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.655104 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:54.655717 28157 raft_consensus.cc:3058] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:54.659535 28200 tablet_bootstrap.cc:654] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:54.662016 28157 raft_consensus.cc:2466] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:58:54.663416 27975 leader_election.cc:304] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:58:54.664494 28487 raft_consensus.cc:2802] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:54.665117 28487 raft_consensus.cc:695] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:58:54.663331 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:54.665937 28487 consensus_queue.cc:237] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.667294 28376 leader_election.cc:290] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:54.668049 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:58:54.669817 28083 raft_consensus.cc:3058] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:54.670614 28202 ts_tablet_manager.cc:1428] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.033s user 0.007s sys 0.000s
I20250905 19:58:54.672863 28202 tablet_bootstrap.cc:492] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:54.674742 28200 tablet_bootstrap.cc:492] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:54.675216 28200 ts_tablet_manager.cc:1397] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.003s
I20250905 19:58:54.677580 28083 raft_consensus.cc:2466] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:58:54.677739 28200 raft_consensus.cc:357] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.697324 28200 raft_consensus.cc:383] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:54.697629 28200 raft_consensus.cc:738] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:54.698261 28200 consensus_queue.cc:260] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.700429 28200 ts_tablet_manager.cc:1428] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20250905 19:58:54.687253 28202 tablet_bootstrap.cc:654] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:54.691900 27905 catalog_manager.cc:5582] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:54.707412 28200 tablet_bootstrap.cc:492] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:54.713629 28200 tablet_bootstrap.cc:654] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:54.694674 28050 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:54.696511 28393 consensus_queue.cc:1035] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:54.721576 28436 raft_consensus.cc:3058] T c0889204295e4daa99d81e5b2a45ac1c P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:58:54.736457 27973 leader_election.cc:343] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:54.737052 27973 leader_election.cc:304] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:54.744194 28436 raft_consensus.cc:2747] T c0889204295e4daa99d81e5b2a45ac1c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:54.744416 28200 tablet_bootstrap.cc:492] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:54.744997 28200 ts_tablet_manager.cc:1397] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.010s
I20250905 19:58:54.748338 28200 raft_consensus.cc:357] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.749251 28200 raft_consensus.cc:383] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:54.749569 28200 raft_consensus.cc:738] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:54.750511 28393 raft_consensus.cc:2747] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:54.753806 28201 tablet_bootstrap.cc:492] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:54.754261 28201 ts_tablet_manager.cc:1397] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.199s user 0.026s sys 0.047s
I20250905 19:58:54.756917 28201 raft_consensus.cc:357] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.757625 28201 raft_consensus.cc:383] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:54.757905 28201 raft_consensus.cc:738] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:54.758541 28201 consensus_queue.cc:260] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.760792 28201 ts_tablet_manager.cc:1428] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250905 19:58:54.761730 28201 tablet_bootstrap.cc:492] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:54.772652 28375 raft_consensus.cc:491] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:54.773231 28375 raft_consensus.cc:513] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.774091 28201 tablet_bootstrap.cc:654] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:54.775409 28375 leader_election.cc:290] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 19:58:54.777659 27956 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250905 19:58:54.778105 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:58:54.779588 28122 leader_election.cc:343] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:54.782202 28485 consensus_queue.cc:1035] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:54.782526 28200 consensus_queue.cc:260] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.785692 28200 ts_tablet_manager.cc:1428] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.040s user 0.006s sys 0.000s
I20250905 19:58:54.785934 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:54.789494 28504 raft_consensus.cc:491] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:58:54.789793 28125 leader_election.cc:343] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:54.790077 28504 raft_consensus.cc:513] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.790325 28125 leader_election.cc:304] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:54.791186 28375 raft_consensus.cc:2747] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:54.792459 28504 leader_election.cc:290] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:54.793568 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:58:54.794837 28051 leader_election.cc:343] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:54.799266 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:54.798751 27906 catalog_manager.cc:5582] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:54.800191 28157 raft_consensus.cc:2466] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:54.801142 28200 tablet_bootstrap.cc:492] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:54.802163 28202 tablet_bootstrap.cc:492] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:54.802678 28202 ts_tablet_manager.cc:1397] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.130s user 0.025s sys 0.019s
I20250905 19:58:54.805382 28202 raft_consensus.cc:357] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.806088 28202 raft_consensus.cc:383] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:54.806358 28202 raft_consensus.cc:738] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:54.806988 28202 consensus_queue.cc:260] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.808827 28200 tablet_bootstrap.cc:654] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:54.809221 28202 ts_tablet_manager.cc:1428] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250905 19:58:54.810578 28202 tablet_bootstrap.cc:492] T 1b18b7149ff3439a8dde09149cf336b3 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:54.816399 28050 leader_election.cc:304] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:54.819235 28504 raft_consensus.cc:2802] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:54.819631 28504 raft_consensus.cc:491] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:54.820031 28201 tablet_bootstrap.cc:492] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:54.819995 28504 raft_consensus.cc:3058] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:54.820572 28201 ts_tablet_manager.cc:1397] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.059s user 0.016s sys 0.003s
I20250905 19:58:54.823276 28201 raft_consensus.cc:357] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.824007 28201 raft_consensus.cc:383] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:54.824263 28201 raft_consensus.cc:738] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:54.824849 28201 consensus_queue.cc:260] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.827266 28201 ts_tablet_manager.cc:1428] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250905 19:58:54.827013 28375 consensus_queue.cc:1035] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:54.828310 28201 tablet_bootstrap.cc:492] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:54.817596 28202 tablet_bootstrap.cc:654] T 1b18b7149ff3439a8dde09149cf336b3 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:54.841746 28201 tablet_bootstrap.cc:654] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:54.832351 28504 raft_consensus.cc:513] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.847400 28375 raft_consensus.cc:491] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:54.848318 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:54.848268 28375 raft_consensus.cc:513] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:58:54.850279 28051 leader_election.cc:343] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:54.853976 28488 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:54.854156 28504 leader_election.cc:290] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:54.854307 28202 tablet_bootstrap.cc:492] T 1b18b7149ff3439a8dde09149cf336b3 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:54.854000 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:54.936645 28202 ts_tablet_manager.cc:1397] T 1b18b7149ff3439a8dde09149cf336b3 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.126s user 0.029s sys 0.027s
I20250905 19:58:54.892946 28373 consensus_queue.cc:1035] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.044s
I20250905 19:58:54.854625 28488 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.922842 28485 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:54.947468 28202 raft_consensus.cc:357] T 1b18b7149ff3439a8dde09149cf336b3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.948256 28202 raft_consensus.cc:383] T 1b18b7149ff3439a8dde09149cf336b3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:54.947919 28485 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.948552 28202 raft_consensus.cc:738] T 1b18b7149ff3439a8dde09149cf336b3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:54.949256 28202 consensus_queue.cc:260] T 1b18b7149ff3439a8dde09149cf336b3 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.867887 28375 leader_election.cc:290] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:54.936791 28157 raft_consensus.cc:3058] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:54.868893 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:54.875272 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:54.900317 28376 consensus_queue.cc:1035] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.039s
I20250905 19:58:54.917467 28436 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:54.956965 28488 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:54.957633 28485 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:54.954355 28202 ts_tablet_manager.cc:1428] T 1b18b7149ff3439a8dde09149cf336b3 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.017s user 0.005s sys 0.001s
I20250905 19:58:54.958926 28202 tablet_bootstrap.cc:492] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:54.959565 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:54.960465 28159 raft_consensus.cc:2391] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 1: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:58:54.956621 28436 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.963110 28157 raft_consensus.cc:2466] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:54.966513 28436 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:58:54.968387 28122 leader_election.cc:343] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:54.969497 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:54.970577 28010 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:54.972235 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:54.964668 28202 tablet_bootstrap.cc:654] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:54.973284 28085 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 0.
I20250905 19:58:54.974767 28122 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:54.975950 28491 raft_consensus.cc:2802] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:54.976078 28051 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:54.976614 28491 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:54.977113 28501 raft_consensus.cc:2802] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:54.977325 28491 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:54.977644 28501 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:54.978209 28501 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:54.988261 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:54.990515 28202 tablet_bootstrap.cc:492] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:54.991093 28202 ts_tablet_manager.cc:1397] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.000s
I20250905 19:58:54.988446 28050 leader_election.cc:304] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:54.992599 28435 raft_consensus.cc:2802] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:54.993228 28435 raft_consensus.cc:695] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:58:54.993458 28202 raft_consensus.cc:357] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.994371 28435 consensus_queue.cc:237] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:54.994398 28202 raft_consensus.cc:383] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:54.999735 28202 raft_consensus.cc:738] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:55.000453 28202 consensus_queue.cc:260] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.001356 28200 tablet_bootstrap.cc:492] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:55.001992 28200 ts_tablet_manager.cc:1397] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.201s user 0.032s sys 0.058s
I20250905 19:58:55.002823 28202 ts_tablet_manager.cc:1428] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20250905 19:58:55.004027 28202 tablet_bootstrap.cc:492] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:55.004966 28200 raft_consensus.cc:357] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.007683 28200 raft_consensus.cc:383] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:55.008047 28200 raft_consensus.cc:738] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:55.008739 28200 consensus_queue.cc:260] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.010488 28200 ts_tablet_manager.cc:1428] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250905 19:58:55.013368 28201 tablet_bootstrap.cc:492] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:55.013868 28201 ts_tablet_manager.cc:1397] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.186s user 0.049s sys 0.022s
I20250905 19:58:55.014513 28376 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:55.015252 28200 tablet_bootstrap.cc:492] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:55.015022 28376 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.017158 28376 leader_election.cc:290] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:55.018731 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:55.019213 28202 tablet_bootstrap.cc:654] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
W20250905 19:58:55.020296 27975 leader_election.cc:343] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:55.022305 28200 tablet_bootstrap.cc:654] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:55.023831 28501 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.031045 28200 tablet_bootstrap.cc:492] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:55.031608 28200 ts_tablet_manager.cc:1397] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.000s
I20250905 19:58:55.033141 28202 tablet_bootstrap.cc:492] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:55.033627 28202 ts_tablet_manager.cc:1397] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.005s
I20250905 19:58:55.034395 28200 raft_consensus.cc:357] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.035156 28200 raft_consensus.cc:383] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:55.026242 28501 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:55.035451 28200 raft_consensus.cc:738] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:55.035969 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:55.036159 28200 consensus_queue.cc:260] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.036863 28202 raft_consensus.cc:357] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.039395 28200 ts_tablet_manager.cc:1428] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250905 19:58:55.040407 28200 tablet_bootstrap.cc:492] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:55.037521 28202 raft_consensus.cc:383] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:55.042207 28202 raft_consensus.cc:738] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:55.042690 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:55.042908 28202 consensus_queue.cc:260] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.043351 28004 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:55.045277 28202 ts_tablet_manager.cc:1428] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250905 19:58:55.045075 28488 consensus_queue.cc:1035] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:55.046295 28202 tablet_bootstrap.cc:492] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:55.046365 28200 tablet_bootstrap.cc:654] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:55.049078 28201 raft_consensus.cc:357] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.049778 28201 raft_consensus.cc:383] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:55.050069 28201 raft_consensus.cc:738] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:55.050745 28201 consensus_queue.cc:260] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.051079 28004 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:55.056358 28201 ts_tablet_manager.cc:1428] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.042s user 0.008s sys 0.000s
I20250905 19:58:55.058645 28201 tablet_bootstrap.cc:492] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
W20250905 19:58:55.067262 28125 leader_election.cc:343] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:55.067921 28125 leader_election.cc:304] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:55.069031 28373 raft_consensus.cc:2747] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:55.073695 28201 tablet_bootstrap.cc:654] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:55.078169 28202 tablet_bootstrap.cc:654] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:55.087857 28202 tablet_bootstrap.cc:492] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:55.088353 28202 ts_tablet_manager.cc:1397] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.042s user 0.008s sys 0.004s
I20250905 19:58:55.090911 28202 raft_consensus.cc:357] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.091604 28202 raft_consensus.cc:383] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:55.091883 28202 raft_consensus.cc:738] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:55.092562 28202 consensus_queue.cc:260] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.099090 28200 tablet_bootstrap.cc:492] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:55.099622 28200 ts_tablet_manager.cc:1397] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.059s user 0.020s sys 0.009s
I20250905 19:58:55.101730 28202 ts_tablet_manager.cc:1428] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20250905 19:58:55.102722 28202 tablet_bootstrap.cc:492] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:55.102310 28200 raft_consensus.cc:357] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.103206 28200 raft_consensus.cc:383] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:55.103498 28200 raft_consensus.cc:738] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:55.104121 28200 consensus_queue.cc:260] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.104835 28201 tablet_bootstrap.cc:492] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:55.105373 28201 ts_tablet_manager.cc:1397] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.047s user 0.012s sys 0.006s
I20250905 19:58:55.106417 28200 ts_tablet_manager.cc:1428] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250905 19:58:55.107409 28200 tablet_bootstrap.cc:492] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:55.107244 27908 catalog_manager.cc:5582] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:55.108368 28201 raft_consensus.cc:357] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.109066 28201 raft_consensus.cc:383] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:55.109221 28202 tablet_bootstrap.cc:654] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:55.109378 28201 raft_consensus.cc:738] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:55.110183 28201 consensus_queue.cc:260] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.112736 28201 ts_tablet_manager.cc:1428] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250905 19:58:55.114521 28201 tablet_bootstrap.cc:492] T 4f2279535926437ba84ae91f2b83b3d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:55.115237 28200 tablet_bootstrap.cc:654] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:55.123956 28201 tablet_bootstrap.cc:654] T 4f2279535926437ba84ae91f2b83b3d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:55.127305 28200 tablet_bootstrap.cc:492] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:55.127817 28200 ts_tablet_manager.cc:1397] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.011s
I20250905 19:58:55.133433 28373 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:55.138017 28200 raft_consensus.cc:357] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.138824 28200 raft_consensus.cc:383] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:55.139148 28200 raft_consensus.cc:738] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:55.139916 28200 consensus_queue.cc:260] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.134265 28373 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.142184 28200 ts_tablet_manager.cc:1428] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.014s user 0.008s sys 0.002s
I20250905 19:58:55.143045 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:55.145892 28010 raft_consensus.cc:2391] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 1: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:58:55.146165 28200 tablet_bootstrap.cc:492] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:55.150954 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:55.153687 28051 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:58:55.153605 28491 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.197969 28081 raft_consensus.cc:3058] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:55.199297 28157 raft_consensus.cc:2391] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 1: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:58:55.199770 28504 raft_consensus.cc:491] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:55.200263 28504 raft_consensus.cc:513] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.158550 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:55.202143 28504 leader_election.cc:290] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:55.177567 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:55.202975 28004 raft_consensus.cc:2466] T 8a8d8393352b4214a20c1683852d4fdd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 0.
I20250905 19:58:55.172276 28373 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:55.155634 28201 tablet_bootstrap.cc:492] T 4f2279535926437ba84ae91f2b83b3d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:55.204592 28201 ts_tablet_manager.cc:1397] T 4f2279535926437ba84ae91f2b83b3d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.090s user 0.034s sys 0.013s
I20250905 19:58:55.188442 28488 consensus_queue.cc:1035] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20250905 19:58:55.202167 28082 raft_consensus.cc:2466] T 1b18b7149ff3439a8dde09149cf336b3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 0.
I20250905 19:58:55.207882 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:55.208910 28491 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 19:58:55.209748 28050 leader_election.cc:343] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:55.209919 28201 raft_consensus.cc:357] T 4f2279535926437ba84ae91f2b83b3d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.211127 28201 raft_consensus.cc:383] T 4f2279535926437ba84ae91f2b83b3d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:55.211556 28201 raft_consensus.cc:738] T 4f2279535926437ba84ae91f2b83b3d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:55.154915 28435 raft_consensus.cc:2802] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:55.213693 28435 raft_consensus.cc:695] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:58:55.214640 28435 consensus_queue.cc:237] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.216450 28200 tablet_bootstrap.cc:654] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:55.173988 28375 raft_consensus.cc:491] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:55.212513 28201 consensus_queue.cc:260] T 4f2279535926437ba84ae91f2b83b3d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.220541 27973 leader_election.cc:304] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:55.222383 28494 raft_consensus.cc:3058] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:55.224975 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:55.226554 28202 tablet_bootstrap.cc:492] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:55.227070 28202 ts_tablet_manager.cc:1397] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.125s user 0.042s sys 0.055s
I20250905 19:58:55.218617 28375 raft_consensus.cc:513] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.232414 28201 ts_tablet_manager.cc:1428] T 4f2279535926437ba84ae91f2b83b3d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.027s user 0.008s sys 0.000s
I20250905 19:58:55.233407 27973 leader_election.cc:304] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:58:55.234112 28375 leader_election.cc:290] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:55.233351 28494 raft_consensus.cc:2747] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:55.235378 28485 consensus_queue.cc:1035] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:55.234328 28376 raft_consensus.cc:2802] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:55.236284 28376 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:55.236677 28376 raft_consensus.cc:3058] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:55.237362 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:55.238075 28004 raft_consensus.cc:2466] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:55.243611 28376 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.244849 28494 consensus_queue.cc:1035] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:55.246733 28376 leader_election.cc:290] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:55.247704 28125 leader_election.cc:304] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:58:55.249608 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 19:58:55.251559 27975 leader_election.cc:343] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:55.252229 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:55.253013 28004 raft_consensus.cc:2391] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 1: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:58:55.254972 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:55.258709 28200 tablet_bootstrap.cc:492] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:55.259218 28200 ts_tablet_manager.cc:1397] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.113s user 0.011s sys 0.002s
I20250905 19:58:55.262847 28200 raft_consensus.cc:357] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.263607 28200 raft_consensus.cc:383] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:55.263880 28200 raft_consensus.cc:738] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:55.265396 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:58:55.266112 28082 raft_consensus.cc:3058] T 1b18b7149ff3439a8dde09149cf336b3 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:55.264511 28200 consensus_queue.cc:260] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.269361 28200 ts_tablet_manager.cc:1428] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
W20250905 19:58:55.271145 28122 leader_election.cc:343] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:55.271449 28200 tablet_bootstrap.cc:492] T fba42101cd094077aa85e3f9186366e7 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:55.271016 28376 consensus_queue.cc:1035] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:55.273674 28082 raft_consensus.cc:2466] T 1b18b7149ff3439a8dde09149cf336b3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:58:55.266098 28491 raft_consensus.cc:2802] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:55.275091 28491 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:55.275430 28491 raft_consensus.cc:3058] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:55.235244 28201 tablet_bootstrap.cc:492] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:55.278790 28200 tablet_bootstrap.cc:654] T fba42101cd094077aa85e3f9186366e7 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:55.233172 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:58:55.234777 28202 raft_consensus.cc:357] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.279296 28202 raft_consensus.cc:383] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:55.279589 28202 raft_consensus.cc:738] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:55.280220 28202 consensus_queue.cc:260] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.284017 28202 ts_tablet_manager.cc:1428] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.057s user 0.007s sys 0.000s
I20250905 19:58:55.283677 27907 catalog_manager.cc:5582] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:55.284996 28202 tablet_bootstrap.cc:492] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
W20250905 19:58:55.286022 28125 leader_election.cc:343] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:55.286845 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:55.291882 28051 leader_election.cc:304] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:58:55.294847 28501 raft_consensus.cc:2802] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:55.298367 28501 raft_consensus.cc:491] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:55.309146 27973 leader_election.cc:304] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:58:55.314183 28436 consensus_queue.cc:1035] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:55.310228 28501 raft_consensus.cc:3058] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:55.315289 28376 raft_consensus.cc:2802] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:55.315836 28376 raft_consensus.cc:695] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:58:55.318024 28376 consensus_queue.cc:237] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.357468 28375 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:55.357676 28436 raft_consensus.cc:491] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:55.393582 28202 tablet_bootstrap.cc:654] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
W20250905 19:58:55.428722 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 80030.4us
I20250905 19:58:55.394956 28375 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.400826 28436 raft_consensus.cc:513] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.446132 28589 consensus_queue.cc:1035] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20250905 19:58:55.416211 28491 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.468430 28491 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:55.463207 28488 raft_consensus.cc:491] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:55.434991 28373 raft_consensus.cc:491] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:55.471033 28373 raft_consensus.cc:513] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.472087 28436 leader_election.cc:290] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:55.469787 28488 raft_consensus.cc:513] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.488279 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:55.488986 28156 raft_consensus.cc:2466] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:55.484350 28435 raft_consensus.cc:491] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:55.491779 28050 leader_election.cc:304] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 19:58:55.492164 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 48732.5us
I20250905 19:58:55.479435 28373 leader_election.cc:290] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:55.495400 28376 raft_consensus.cc:491] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:55.496083 28376 raft_consensus.cc:513] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.468523 28501 raft_consensus.cc:513] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.490123 28435 raft_consensus.cc:513] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:58:55.507983 28582 log.cc:927] Time spent T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.062s user 0.003s sys 0.000s
I20250905 19:58:55.508716 28375 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:55.493664 28491 raft_consensus.cc:491] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:55.512651 28201 tablet_bootstrap.cc:654] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:55.490151 28488 leader_election.cc:290] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:55.527894 28122 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:55.533610 28376 leader_election.cc:290] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:58:55.535506 28122 leader_election.cc:343] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:55.535897 28375 raft_consensus.cc:2747] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250905 19:58:55.538283 28393 raft_consensus.cc:491] T 4f2279535926437ba84ae91f2b83b3d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:55.538841 28393 raft_consensus.cc:513] T 4f2279535926437ba84ae91f2b83b3d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.535995 28122 leader_election.cc:304] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:55.540045 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:55.549628 28393 leader_election.cc:290] T 4f2279535926437ba84ae91f2b83b3d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:55.549629 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:55.551124 28156 raft_consensus.cc:2466] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 0.
I20250905 19:58:55.551527 28501 leader_election.cc:290] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:55.552464 28488 raft_consensus.cc:2747] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:55.553074 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:55.530548 28491 raft_consensus.cc:513] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.556119 28491 leader_election.cc:290] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:55.534214 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:55.557807 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:55.558728 28157 raft_consensus.cc:2391] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 1: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:58:55.562908 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 19:58:55.565107 28050 leader_election.cc:343] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:55.566618 27975 leader_election.cc:304] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:58:55.584219 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:58:55.569075 28125 leader_election.cc:343] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:55.574110 28435 leader_election.cc:290] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:55.526185 28504 raft_consensus.cc:2802] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:55.584864 27975 leader_election.cc:304] T 4f2279535926437ba84ae91f2b83b3d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:55.588780 28376 raft_consensus.cc:3058] T 4f2279535926437ba84ae91f2b83b3d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:55.585248 28157 raft_consensus.cc:2466] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:55.587072 28504 raft_consensus.cc:491] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:55.590152 28504 raft_consensus.cc:3058] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:55.593708 28050 leader_election.cc:304] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:55.587350 28494 raft_consensus.cc:2802] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:55.595063 28080 raft_consensus.cc:2466] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 0.
I20250905 19:58:55.631286 28571 raft_consensus.cc:2802] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:55.614812 28494 raft_consensus.cc:491] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:55.631304 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:55.638089 28494 raft_consensus.cc:3058] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:55.687600 28005 raft_consensus.cc:3058] T 8a8d8393352b4214a20c1683852d4fdd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:55.706317 28202 tablet_bootstrap.cc:492] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:55.706979 28202 ts_tablet_manager.cc:1397] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.422s user 0.008s sys 0.025s
I20250905 19:58:55.718247 28200 tablet_bootstrap.cc:492] T fba42101cd094077aa85e3f9186366e7 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:55.718971 28200 ts_tablet_manager.cc:1397] T fba42101cd094077aa85e3f9186366e7 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.449s user 0.020s sys 0.007s
I20250905 19:58:55.744717 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:55.726064 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:55.771648 28202 raft_consensus.cc:357] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.772670 28202 raft_consensus.cc:383] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:55.773010 28202 raft_consensus.cc:738] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:55.780253 28201 tablet_bootstrap.cc:492] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:55.780807 28201 ts_tablet_manager.cc:1397] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.546s user 0.040s sys 0.055s
I20250905 19:58:55.778087 28376 raft_consensus.cc:2747] T 4f2279535926437ba84ae91f2b83b3d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:55.790737 28202 consensus_queue.cc:260] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.797915 28571 raft_consensus.cc:491] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:55.798566 28200 raft_consensus.cc:357] T fba42101cd094077aa85e3f9186366e7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.800521 28200 raft_consensus.cc:383] T fba42101cd094077aa85e3f9186366e7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:55.800805 28200 raft_consensus.cc:738] T fba42101cd094077aa85e3f9186366e7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:55.801965 28200 consensus_queue.cc:260] T fba42101cd094077aa85e3f9186366e7 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:58:55.803714 28122 leader_election.cc:343] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:55.804373 28122 leader_election.cc:304] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:55.823805 28202 ts_tablet_manager.cc:1428] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.116s user 0.007s sys 0.002s
I20250905 19:58:55.798403 28571 raft_consensus.cc:3058] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:55.799587 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:58:55.828763 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:55.830694 28005 raft_consensus.cc:2466] T 8a8d8393352b4214a20c1683852d4fdd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 19:58:55.817329 28373 raft_consensus.cc:2747] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:55.801554 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:55.798913 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:55.812891 28589 raft_consensus.cc:491] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:55.857493 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:55.864784 28201 raft_consensus.cc:357] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.870581 28201 raft_consensus.cc:383] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:55.870960 28201 raft_consensus.cc:738] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:55.874110 28201 consensus_queue.cc:260] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.877348 28373 raft_consensus.cc:491] T fba42101cd094077aa85e3f9186366e7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:55.884925 28373 raft_consensus.cc:513] T fba42101cd094077aa85e3f9186366e7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.888696 28200 ts_tablet_manager.cc:1428] T fba42101cd094077aa85e3f9186366e7 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.169s user 0.007s sys 0.000s
W20250905 19:58:55.900586 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 86652.8us
W20250905 19:58:55.913002 28125 leader_election.cc:343] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:55.918299 28125 leader_election.cc:304] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:58:55.919360 28201 ts_tablet_manager.cc:1428] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.138s user 0.008s sys 0.001s
I20250905 19:58:55.869678 28589 raft_consensus.cc:513] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.937454 28613 consensus_queue.cc:1035] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20250905 19:58:55.940145 28611 raft_consensus.cc:2802] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:55.948009 28589 leader_election.cc:290] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:55.926049 28494 raft_consensus.cc:513] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.955773 28494 leader_election.cc:290] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:58:55.957442 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 87835.2us
W20250905 19:58:55.961500 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 46966.1us
W20250905 19:58:55.958701 28122 leader_election.cc:343] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:55.962785 28122 leader_election.cc:304] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
W20250905 19:58:55.963761 28122 leader_election.cc:343] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:58:55.964411 28122 leader_election.cc:343] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:58:55.964841 28122 leader_election.cc:343] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:55.940945 28611 raft_consensus.cc:695] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
W20250905 19:58:55.974802 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 52149.4us
I20250905 19:58:55.971318 28375 raft_consensus.cc:2747] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:58:55.988399 27975 consensus_peers.cc:489] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 -> Peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Couldn't send request to peer 26b2f874b8e54f08b01a9ed00d5cce05. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:58:55.972196 28504 raft_consensus.cc:513] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.961231 28571 raft_consensus.cc:513] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:55.999235 28373 leader_election.cc:290] T fba42101cd094077aa85e3f9186366e7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:56.106983 28571 leader_election.cc:290] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:56.056707 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:56.058782 28611 consensus_queue.cc:237] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:56.143787 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:56.135705 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:56.131104 28151 raft_consensus.cc:2466] T fba42101cd094077aa85e3f9186366e7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
W20250905 19:58:56.152395 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 156564us
W20250905 19:58:56.153491 28051 leader_election.cc:343] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:56.158574 28050 leader_election.cc:304] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:56.162587 28613 raft_consensus.cc:2802] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:56.163044 28613 raft_consensus.cc:491] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:56.163421 28613 raft_consensus.cc:3058] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:56.158188 28156 raft_consensus.cc:3058] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:56.173943 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:56.177847 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:56.178609 28158 raft_consensus.cc:3058] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:56.167107 28504 leader_election.cc:290] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:56.162626 28571 consensus_queue.cc:1035] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
W20250905 19:58:56.214066 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 97192.9us
W20250905 19:58:56.219008 28051 leader_election.cc:343] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:56.227487 28612 raft_consensus.cc:491] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:56.228231 28612 raft_consensus.cc:513] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:56.245003 28156 raft_consensus.cc:2466] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:56.240547 28612 leader_election.cc:290] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:56.250396 28613 raft_consensus.cc:513] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:56.253608 28613 leader_election.cc:290] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:56.290619 28083 raft_consensus.cc:2391] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 1: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
W20250905 19:58:56.291043 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 63638.9us
I20250905 19:58:56.303645 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:56.305373 28003 raft_consensus.cc:2391] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 1: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:58:56.307133 28158 raft_consensus.cc:2466] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:56.311309 28050 leader_election.cc:304] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:56.319584 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:56.324116 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:56.323714 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:58:56.325580 28125 leader_election.cc:343] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:56.326498 28125 leader_election.cc:304] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:56.326674 28008 raft_consensus.cc:2466] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 0.
I20250905 19:58:56.328619 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:58:56.330037 28078 raft_consensus.cc:2391] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 1: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:58:56.331655 28007 raft_consensus.cc:3058] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:56.341092 28435 consensus_queue.cc:1035] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:56.346519 28501 raft_consensus.cc:2802] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:56.353205 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:56.353483 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:56.354826 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:56.368434 28007 raft_consensus.cc:2466] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:56.371157 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:56.343375 28393 consensus_queue.cc:1035] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20250905 19:58:56.386108 28157 raft_consensus.cc:2391] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 1: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:58:56.386912 28010 raft_consensus.cc:2391] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 1: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:58:56.394320 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:56.403192 28004 raft_consensus.cc:3058] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:56.396282 28201 tablet_bootstrap.cc:492] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
W20250905 19:58:56.398268 27956 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250905 19:58:56.424015 28125 leader_election.cc:304] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:56.425527 28373 raft_consensus.cc:2802] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:56.426033 28373 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:56.426527 28373 raft_consensus.cc:3058] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:56.428784 28202 tablet_bootstrap.cc:492] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:56.436952 28004 raft_consensus.cc:2466] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:56.438097 28501 raft_consensus.cc:695] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:58:56.439055 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:56.439378 28501 consensus_queue.cc:237] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:56.444047 27906 catalog_manager.cc:5582] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:56.447256 28200 tablet_bootstrap.cc:492] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:56.454783 28592 raft_consensus.cc:2747] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:58:56.485532 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 174743us
W20250905 19:58:56.494854 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 71150.1us
W20250905 19:58:56.498621 28125 leader_election.cc:343] T fba42101cd094077aa85e3f9186366e7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:56.507686 28050 leader_election.cc:304] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:56.510761 27907 catalog_manager.cc:5582] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20250905 19:58:56.521843 28201 tablet_bootstrap.cc:654] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:56.528858 28592 raft_consensus.cc:491] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:58:56.529548 28592 raft_consensus.cc:513] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
W20250905 19:58:56.531929 28051 leader_election.cc:343] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:58:56.533932 28051 leader_election.cc:343] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:56.535477 28592 leader_election.cc:290] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:56.536609 28200 tablet_bootstrap.cc:654] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:56.523285 28501 consensus_queue.cc:1035] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20250905 19:58:56.536697 28373 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:56.547116 28051 leader_election.cc:304] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:58:56.548066 28373 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:56.553390 28122 leader_election.cc:304] T fba42101cd094077aa85e3f9186366e7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:56.554929 28366 raft_consensus.cc:3058] T fba42101cd094077aa85e3f9186366e7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:56.556867 28202 tablet_bootstrap.cc:654] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:56.557229 27975 leader_election.cc:304] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:56.563985 28501 raft_consensus.cc:2802] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:56.564105 28586 raft_consensus.cc:2747] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250905 19:58:56.564767 28501 raft_consensus.cc:695] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:58:56.569406 28366 raft_consensus.cc:2747] T fba42101cd094077aa85e3f9186366e7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:56.565848 28501 consensus_queue.cc:237] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:56.584627 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:56.585532 28159 raft_consensus.cc:2466] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:56.585367 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:56.586521 28006 raft_consensus.cc:3058] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:56.588935 28635 raft_consensus.cc:2802] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:56.589350 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:56.589534 28635 raft_consensus.cc:695] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:58:56.592970 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:56.590582 28635 consensus_queue.cc:237] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:56.718300 28050 leader_election.cc:304] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:56.720263 28636 raft_consensus.cc:2802] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:56.721158 28636 raft_consensus.cc:491] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:56.721647 28636 raft_consensus.cc:3058] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:56.721818 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:58:56.679940 28006 raft_consensus.cc:2466] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 19:58:56.730209 28200 tablet_bootstrap.cc:492] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:56.734519 28200 ts_tablet_manager.cc:1397] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.288s user 0.030s sys 0.015s
W20250905 19:58:56.736402 28122 leader_election.cc:343] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:56.616153 28155 raft_consensus.cc:2466] T fba42101cd094077aa85e3f9186366e7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:56.609227 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:56.749938 28200 raft_consensus.cc:357] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:56.750769 28200 raft_consensus.cc:383] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:56.751075 28200 raft_consensus.cc:738] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:56.751842 28200 consensus_queue.cc:260] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:56.770295 28122 leader_election.cc:304] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:56.779438 28125 leader_election.cc:304] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:56.790918 28050 leader_election.cc:304] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 19:58:56.795154 28122 consensus_peers.cc:489] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 -> Peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Couldn't send request to peer e4413ae5e0b64b52a6f360ac580239b9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:58:56.801460 28488 raft_consensus.cc:2802] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:56.802023 28488 raft_consensus.cc:695] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:58:56.803143 28488 consensus_queue.cc:237] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:56.806676 28588 raft_consensus.cc:2747] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:58:56.807554 28588 consensus_queue.cc:1035] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:56.817423 28593 raft_consensus.cc:2802] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:56.822389 28593 raft_consensus.cc:695] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:58:56.823513 28593 consensus_queue.cc:237] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:56.836671 28202 tablet_bootstrap.cc:492] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:56.838727 28202 ts_tablet_manager.cc:1397] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.411s user 0.013s sys 0.016s
I20250905 19:58:56.841619 28200 ts_tablet_manager.cc:1428] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.106s user 0.008s sys 0.000s
I20250905 19:58:56.880666 28200 tablet_bootstrap.cc:492] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:56.886572 28636 raft_consensus.cc:513] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:56.900460 28201 tablet_bootstrap.cc:492] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:56.900909 28201 ts_tablet_manager.cc:1397] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.505s user 0.037s sys 0.011s
I20250905 19:58:56.901229 28202 raft_consensus.cc:357] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:56.903909 28202 raft_consensus.cc:383] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:56.904188 28202 raft_consensus.cc:738] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:56.904744 28202 consensus_queue.cc:260] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:58:56.920161 28654 log.cc:927] Time spent T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.058s user 0.000s sys 0.001s
I20250905 19:58:56.920589 28501 consensus_queue.cc:1035] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250905 19:58:56.931146 28646 log.cc:927] Time spent T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.149s user 0.005s sys 0.000s
I20250905 19:58:56.920804 28201 raft_consensus.cc:357] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:56.932054 28201 raft_consensus.cc:383] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:56.932394 28201 raft_consensus.cc:738] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:56.921846 28202 ts_tablet_manager.cc:1428] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.083s user 0.006s sys 0.001s
I20250905 19:58:56.932991 28201 consensus_queue.cc:260] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:58:56.951054 28651 log.cc:927] Time spent T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.056s user 0.000s sys 0.001s
I20250905 19:58:56.953775 28201 ts_tablet_manager.cc:1428] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.053s user 0.005s sys 0.002s
I20250905 19:58:56.954890 28201 tablet_bootstrap.cc:492] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:56.959998 28501 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:56.960721 28202 tablet_bootstrap.cc:492] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:56.960600 28501 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:56.963776 28501 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:56.967311 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:56.968782 28004 raft_consensus.cc:2391] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 1: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:58:56.967767 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:56.969477 28006 raft_consensus.cc:3058] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:56.970875 28636 leader_election.cc:290] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:56.970317 28200 tablet_bootstrap.cc:654] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:56.981786 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:56.982374 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:56.983052 28155 raft_consensus.cc:3058] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:56.983223 28006 raft_consensus.cc:2466] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:56.991248 27906 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34440 (request call id 87) took 1306 ms. Trace:
I20250905 19:58:56.991739 28201 tablet_bootstrap.cc:654] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:56.994415 28051 leader_election.cc:304] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:56.997504 28636 raft_consensus.cc:2802] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:56.998178 28636 raft_consensus.cc:695] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:58:57.000008 28050 leader_election.cc:304] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:57.000720 28155 raft_consensus.cc:2466] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:57.000844 28501 raft_consensus.cc:3058] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:57.023005 28202 tablet_bootstrap.cc:654] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:57.022775 28635 consensus_queue.cc:1035] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:57.000275 28636 consensus_queue.cc:237] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:56.991807 27906 rpcz_store.cc:276] 0905 19:58:55.685135 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:58:55.685439 (+ 304us) service_pool.cc:225] Handling call
0905 19:58:56.991164 (+1305725us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:58:56.483881 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:58:56.491635 (+ 7754us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:58:56.491717 (+ 82us) write_op.cc:435] Acquired schema lock
0905 19:58:56.491772 (+ 55us) tablet.cc:660] Decoding operations
0905 19:58:56.492407 (+ 635us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:58:56.492588 (+ 181us) write_op.cc:641] Partition lock acquired for write op
0905 19:58:56.492645 (+ 57us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:58:56.505395 (+ 12750us) tablet.cc:699] Row locks acquired
0905 19:58:56.505441 (+ 46us) write_op.cc:260] PREPARE: finished
0905 19:58:56.512978 (+ 7537us) write_op.cc:270] Start()
0905 19:58:56.513169 (+ 191us) write_op.cc:276] Timestamp: P: 1757102336505577 usec, L: 0
0905 19:58:56.513210 (+ 41us) op_driver.cc:348] REPLICATION: starting
0905 19:58:56.516191 (+ 2981us) log.cc:844] Serialized 540 byte log entry
0905 19:58:56.713047 (+196856us) op_driver.cc:464] REPLICATION: finished
0905 19:58:56.781691 (+ 68644us) write_op.cc:301] APPLY: starting
0905 19:58:56.781786 (+ 95us) tablet.cc:1365] starting BulkCheckPresence
0905 19:58:56.781934 (+ 148us) tablet.cc:1368] finished BulkCheckPresence
0905 19:58:56.813443 (+ 31509us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:58:56.828258 (+ 14815us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:58:56.828528 (+ 270us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:58:56.828601 (+ 73us) write_op.cc:312] APPLY: finished
0905 19:58:56.828939 (+ 338us) log.cc:844] Serialized 22 byte log entry
0905 19:58:56.843202 (+ 14263us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:58:56.857584 (+ 14382us) write_op.cc:454] Released schema lock
0905 19:58:56.861271 (+ 3687us) write_op.cc:341] FINISH: Updating metrics
0905 19:58:56.867437 (+ 6166us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:58:56.869171 (+ 1734us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":19265,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":68322,"apply.run_cpu_time_us":4593,"apply.run_wall_time_us":85603,"num_ops":1,"prepare.queue_time_us":29887,"prepare.run_cpu_time_us":48225,"prepare.run_wall_time_us":114414,"raft.queue_time_us":54941,"raft.run_cpu_time_us":14101,"raft.run_wall_time_us":96883,"replication_time_us":199612,"thread_start_us":197808,"threads_started":4,"wal-append.queue_time_us":92087}]]}
I20250905 19:58:57.031513 28501 raft_consensus.cc:2747] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:57.045692 27908 catalog_manager.cc:5582] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20250905 19:58:57.053349 28635 consensus_queue.cc:1035] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20250905 19:58:57.068341 28201 tablet_bootstrap.cc:492] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:57.068127 27908 catalog_manager.cc:5582] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } }
I20250905 19:58:57.069064 28201 ts_tablet_manager.cc:1397] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.114s user 0.010s sys 0.005s
W20250905 19:58:57.084146 28050 consensus_peers.cc:489] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 -> Peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Couldn't send request to peer 26b2f874b8e54f08b01a9ed00d5cce05. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:58:57.089354 28202 tablet_bootstrap.cc:492] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:57.090013 28202 ts_tablet_manager.cc:1397] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.130s user 0.025s sys 0.011s
I20250905 19:58:57.092911 28202 raft_consensus.cc:357] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.095082 28201 raft_consensus.cc:357] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.096252 28201 raft_consensus.cc:383] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:57.096596 28201 raft_consensus.cc:738] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:57.097030 28202 raft_consensus.cc:383] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:57.097527 28202 raft_consensus.cc:738] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:57.097313 28201 consensus_queue.cc:260] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.098186 28202 consensus_queue.cc:260] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.099972 28201 ts_tablet_manager.cc:1428] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.025s user 0.008s sys 0.000s
I20250905 19:58:57.101121 28200 tablet_bootstrap.cc:492] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:57.101649 28200 ts_tablet_manager.cc:1397] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.239s user 0.017s sys 0.008s
I20250905 19:58:57.101114 28201 tablet_bootstrap.cc:492] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:57.105122 28202 ts_tablet_manager.cc:1428] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20250905 19:58:57.108227 28202 tablet_bootstrap.cc:492] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:57.112429 28201 tablet_bootstrap.cc:654] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:57.108407 28200 raft_consensus.cc:357] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.114634 28200 raft_consensus.cc:383] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:57.114964 28200 raft_consensus.cc:738] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:57.115305 27907 catalog_manager.cc:5582] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:57.116207 28200 consensus_queue.cc:260] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.116562 27907 catalog_manager.cc:5582] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:57.119575 28200 ts_tablet_manager.cc:1428] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.018s user 0.007s sys 0.001s
I20250905 19:58:57.120066 28202 tablet_bootstrap.cc:654] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:57.123016 28200 tablet_bootstrap.cc:492] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
I20250905 19:58:57.126287 27907 catalog_manager.cc:5582] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:57.133343 28201 tablet_bootstrap.cc:492] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:57.134066 28201 ts_tablet_manager.cc:1397] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.033s user 0.013s sys 0.003s
I20250905 19:58:57.136245 28202 tablet_bootstrap.cc:492] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:57.136742 28202 ts_tablet_manager.cc:1397] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.004s
I20250905 19:58:57.136865 28201 raft_consensus.cc:357] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.137595 28201 raft_consensus.cc:383] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:57.137893 28201 raft_consensus.cc:738] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:57.138512 28201 consensus_queue.cc:260] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.139849 28202 raft_consensus.cc:357] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.140563 28200 tablet_bootstrap.cc:654] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:57.140610 28202 raft_consensus.cc:383] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:57.141040 28202 raft_consensus.cc:738] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:57.141826 28202 consensus_queue.cc:260] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.143076 28488 raft_consensus.cc:491] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:57.143587 28488 raft_consensus.cc:513] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.144690 28202 ts_tablet_manager.cc:1428] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20250905 19:58:57.145236 28201 ts_tablet_manager.cc:1428] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250905 19:58:57.145653 28488 leader_election.cc:290] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:57.145789 28202 tablet_bootstrap.cc:492] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:57.146278 28201 tablet_bootstrap.cc:492] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:57.147509 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:58:57.150079 28125 leader_election.cc:343] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:57.148779 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:58:57.153061 28122 leader_election.cc:343] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:57.153527 28122 leader_election.cc:304] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:57.153213 28202 tablet_bootstrap.cc:654] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:57.154480 28629 raft_consensus.cc:2747] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:57.155658 28200 tablet_bootstrap.cc:492] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:57.156198 28200 ts_tablet_manager.cc:1397] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.005s
I20250905 19:58:57.158847 28627 raft_consensus.cc:491] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:57.159540 28627 raft_consensus.cc:513] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.161890 28627 leader_election.cc:290] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:57.164544 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:57.164412 28636 consensus_queue.cc:1035] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250905 19:58:57.166074 27973 leader_election.cc:343] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:57.164701 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:57.170063 28200 raft_consensus.cc:357] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.170938 28200 raft_consensus.cc:383] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:57.171243 28200 raft_consensus.cc:738] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:57.171801 28201 tablet_bootstrap.cc:654] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:57.172045 28202 tablet_bootstrap.cc:492] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:57.172492 28202 ts_tablet_manager.cc:1397] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.007s
W20250905 19:58:57.173653 27975 leader_election.cc:343] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:58:57.174124 27975 leader_election.cc:304] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:57.174927 28200 consensus_queue.cc:260] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.175561 28627 raft_consensus.cc:2747] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:57.175915 28616 raft_consensus.cc:491] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:57.175575 28202 raft_consensus.cc:357] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.176602 28202 raft_consensus.cc:383] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:57.176425 28616 raft_consensus.cc:513] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.176893 28202 raft_consensus.cc:738] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:57.177552 28202 consensus_queue.cc:260] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.180193 28202 ts_tablet_manager.cc:1428] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250905 19:58:57.180402 28616 leader_election.cc:290] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:57.181325 28202 tablet_bootstrap.cc:492] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:57.187317 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:57.190034 28202 tablet_bootstrap.cc:654] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:57.189779 28629 consensus_queue.cc:1035] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:57.193776 28200 ts_tablet_manager.cc:1428] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.037s user 0.008s sys 0.000s
I20250905 19:58:57.198609 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:57.207597 28122 leader_election.cc:304] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:57.208760 28616 raft_consensus.cc:2747] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:57.208990 28200 tablet_bootstrap.cc:492] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05: Bootstrap starting.
W20250905 19:58:57.219089 28051 consensus_peers.cc:489] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 -> Peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Couldn't send request to peer cd764bfeebac4cc9ad851cd4a6a7d3b0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:58:57.221493 28636 consensus_queue.cc:1035] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:57.223752 28201 tablet_bootstrap.cc:492] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:57.224339 28201 ts_tablet_manager.cc:1397] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.078s user 0.012s sys 0.002s
I20250905 19:58:57.229521 28201 raft_consensus.cc:357] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.230489 28201 raft_consensus.cc:383] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:57.244637 28201 raft_consensus.cc:738] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:57.246001 28201 consensus_queue.cc:260] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.251798 28616 consensus_queue.cc:1035] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:57.245398 28200 tablet_bootstrap.cc:654] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:57.248106 28202 tablet_bootstrap.cc:492] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:57.252125 28501 consensus_queue.cc:1035] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:57.252346 28201 ts_tablet_manager.cc:1428] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.028s user 0.005s sys 0.000s
I20250905 19:58:57.270587 28202 ts_tablet_manager.cc:1397] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.090s user 0.016s sys 0.015s
I20250905 19:58:57.273929 28202 raft_consensus.cc:357] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.277959 28202 raft_consensus.cc:383] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:57.278445 28202 raft_consensus.cc:738] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:57.274271 28201 tablet_bootstrap.cc:492] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:57.280180 28200 tablet_bootstrap.cc:492] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05: No bootstrap required, opened a new log
I20250905 19:58:57.282405 28200 ts_tablet_manager.cc:1397] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent bootstrapping tablet: real 0.075s user 0.009s sys 0.005s
I20250905 19:58:57.281765 28202 consensus_queue.cc:260] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.286252 28201 tablet_bootstrap.cc:654] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:57.285923 28200 raft_consensus.cc:357] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.286689 28595 raft_consensus.cc:491] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:57.286708 28200 raft_consensus.cc:383] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:57.287096 28202 ts_tablet_manager.cc:1428] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20250905 19:58:57.287263 28200 raft_consensus.cc:738] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Initialized, Role: FOLLOWER
I20250905 19:58:57.287284 28595 raft_consensus.cc:513] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.290197 28202 tablet_bootstrap.cc:492] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:57.292436 28200 consensus_queue.cc:260] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.293901 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:57.294607 28001 raft_consensus.cc:2466] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:57.295018 27907 catalog_manager.cc:5582] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:57.296064 28051 leader_election.cc:304] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:57.297072 28593 raft_consensus.cc:2802] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:57.297452 28593 raft_consensus.cc:491] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:57.297796 28593 raft_consensus.cc:3058] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:57.298309 28595 leader_election.cc:290] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:57.300079 28200 ts_tablet_manager.cc:1428] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent starting tablet: real 0.017s user 0.004s sys 0.002s
I20250905 19:58:57.304651 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:57.305449 28156 raft_consensus.cc:2466] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:57.317592 28201 tablet_bootstrap.cc:492] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:57.318102 28201 ts_tablet_manager.cc:1397] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.044s user 0.011s sys 0.007s
I20250905 19:58:57.320367 28201 raft_consensus.cc:357] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.321010 28201 raft_consensus.cc:383] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:57.321291 28201 raft_consensus.cc:738] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:57.321959 28201 consensus_queue.cc:260] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.323132 28593 raft_consensus.cc:513] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.324879 28201 ts_tablet_manager.cc:1428] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250905 19:58:57.325431 28593 leader_election.cc:290] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:57.304973 28202 tablet_bootstrap.cc:654] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:57.325942 28201 tablet_bootstrap.cc:492] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:57.332291 28201 tablet_bootstrap.cc:654] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:57.336138 28202 tablet_bootstrap.cc:492] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:57.340420 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:57.341146 28001 raft_consensus.cc:3058] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:57.346565 28202 ts_tablet_manager.cc:1397] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.057s user 0.010s sys 0.003s
I20250905 19:58:57.350703 28202 raft_consensus.cc:357] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.351694 28202 raft_consensus.cc:383] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:57.352042 28202 raft_consensus.cc:738] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:57.354499 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:57.355198 28153 raft_consensus.cc:3058] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:57.368073 28202 consensus_queue.cc:260] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.369833 28001 raft_consensus.cc:2466] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:57.370986 28051 leader_election.cc:304] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:57.372537 28595 raft_consensus.cc:2802] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:57.373087 28595 raft_consensus.cc:695] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:58:57.374002 28595 consensus_queue.cc:237] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.376691 28202 ts_tablet_manager.cc:1428] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.030s user 0.006s sys 0.001s
I20250905 19:58:57.391911 28202 tablet_bootstrap.cc:492] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9: Bootstrap starting.
I20250905 19:58:57.392624 28153 raft_consensus.cc:2466] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:57.398659 28201 tablet_bootstrap.cc:492] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:57.399194 28201 ts_tablet_manager.cc:1397] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.073s user 0.039s sys 0.022s
I20250905 19:58:57.401986 28201 raft_consensus.cc:357] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.402756 28201 raft_consensus.cc:383] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:57.403168 28201 raft_consensus.cc:738] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:57.403283 28616 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:57.403792 28616 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.403831 28201 consensus_queue.cc:260] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.405745 28616 leader_election.cc:290] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:57.406241 28201 ts_tablet_manager.cc:1428] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250905 19:58:57.407379 28201 tablet_bootstrap.cc:492] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Bootstrap starting.
I20250905 19:58:57.407855 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:57.411221 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:57.412837 28122 leader_election.cc:304] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:57.413527 28201 tablet_bootstrap.cc:654] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:57.414023 28616 raft_consensus.cc:2747] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:57.418186 28202 tablet_bootstrap.cc:654] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9: Neither blocks nor log segments found. Creating new log.
I20250905 19:58:57.436267 27907 catalog_manager.cc:5582] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:57.450642 28202 tablet_bootstrap.cc:492] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9: No bootstrap required, opened a new log
I20250905 19:58:57.451188 28202 ts_tablet_manager.cc:1397] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9: Time spent bootstrapping tablet: real 0.060s user 0.010s sys 0.004s
I20250905 19:58:57.451711 28201 tablet_bootstrap.cc:492] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0: No bootstrap required, opened a new log
I20250905 19:58:57.452193 28201 ts_tablet_manager.cc:1397] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent bootstrapping tablet: real 0.045s user 0.023s sys 0.014s
I20250905 19:58:57.453509 28202 raft_consensus.cc:357] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.454080 28202 raft_consensus.cc:383] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:57.454304 28202 raft_consensus.cc:738] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Initialized, Role: FOLLOWER
I20250905 19:58:57.454934 28201 raft_consensus.cc:357] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.455065 28202 consensus_queue.cc:260] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.455654 28201 raft_consensus.cc:383] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:58:57.455982 28201 raft_consensus.cc:738] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Initialized, Role: FOLLOWER
I20250905 19:58:57.456565 28201 consensus_queue.cc:260] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.458956 28201 ts_tablet_manager.cc:1428] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250905 19:58:57.459353 28202 ts_tablet_manager.cc:1428] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
W20250905 19:58:57.472785 27956 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250905 19:58:57.485832 28593 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:57.486382 28593 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.488251 28593 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:57.489032 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:57.489298 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:57.490814 28050 leader_election.cc:304] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:58:57.491441 28595 raft_consensus.cc:2747] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:57.525802 28156 raft_consensus.cc:3058] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:57.557617 28593 raft_consensus.cc:491] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:57.558109 28593 raft_consensus.cc:513] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.559898 28593 leader_election.cc:290] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:57.560495 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:57.561249 28616 raft_consensus.cc:491] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:57.561163 28001 raft_consensus.cc:2466] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:57.561703 28616 raft_consensus.cc:513] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.562269 28051 leader_election.cc:304] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:57.563342 28595 raft_consensus.cc:2802] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:57.563710 28595 raft_consensus.cc:491] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:57.564059 28595 raft_consensus.cc:3058] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:57.565471 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:57.566154 28083 raft_consensus.cc:2466] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 0.
I20250905 19:58:57.565307 28616 leader_election.cc:290] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:57.566640 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:57.567377 28001 raft_consensus.cc:2466] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 0.
I20250905 19:58:57.567391 28122 leader_election.cc:304] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:57.568652 28616 raft_consensus.cc:2802] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:57.569064 28616 raft_consensus.cc:491] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:57.565165 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:57.569463 28616 raft_consensus.cc:3058] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:57.569890 28156 raft_consensus.cc:2466] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:57.572970 28595 raft_consensus.cc:513] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.577245 28616 raft_consensus.cc:513] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.578649 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:57.579290 28002 raft_consensus.cc:3058] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:57.580971 28616 leader_election.cc:290] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:58:57.581002 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:57.581759 28001 raft_consensus.cc:3058] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:57.583462 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:58:57.584100 28081 raft_consensus.cc:3058] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:57.586493 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:57.577927 28595 leader_election.cc:290] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:57.591373 28002 raft_consensus.cc:2466] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:57.592375 28153 raft_consensus.cc:3058] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:57.594282 28081 raft_consensus.cc:2466] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 19:58:57.595755 28051 leader_election.cc:304] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:57.596732 28122 leader_election.cc:304] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:57.597611 28616 raft_consensus.cc:2802] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:57.598359 28616 raft_consensus.cc:695] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:58:57.599185 28081 raft_consensus.cc:3058] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:57.599256 28616 consensus_queue.cc:237] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.603040 28153 raft_consensus.cc:2466] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:57.604089 28595 raft_consensus.cc:2802] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:57.606860 28595 raft_consensus.cc:695] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:58:57.609820 28001 raft_consensus.cc:2466] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 19:58:57.610486 28635 consensus_queue.cc:1035] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:57.611137 28595 consensus_queue.cc:237] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.618669 27907 catalog_manager.cc:5582] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20250905 19:58:57.630801 28595 raft_consensus.cc:491] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:57.636345 28595 raft_consensus.cc:513] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.638525 28595 leader_election.cc:290] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:57.639863 28627 raft_consensus.cc:491] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:57.640237 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:57.640383 28627 raft_consensus.cc:513] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.640933 28153 raft_consensus.cc:2466] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:57.642645 28627 leader_election.cc:290] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:57.644160 28050 leader_election.cc:304] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:57.644049 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:57.645766 28635 raft_consensus.cc:2802] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:57.646154 28635 raft_consensus.cc:491] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:57.646493 28635 raft_consensus.cc:3058] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:57.646623 28595 consensus_queue.cc:1035] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:57.647090 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:58:57.642833 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:57.648662 28001 raft_consensus.cc:2466] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:57.650959 27973 leader_election.cc:304] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:58:57.652083 28627 raft_consensus.cc:2747] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:58:57.654381 28635 raft_consensus.cc:513] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.653362 27907 catalog_manager.cc:5582] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:57.660215 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:57.660894 28153 raft_consensus.cc:3058] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:57.659207 28635 leader_election.cc:290] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:57.663924 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:57.664638 28001 raft_consensus.cc:3058] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:57.672315 28001 raft_consensus.cc:2466] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:57.673717 28051 leader_election.cc:304] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:57.674657 28589 raft_consensus.cc:2802] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:57.675190 28589 raft_consensus.cc:695] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:58:57.676052 28589 consensus_queue.cc:237] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.677359 28156 raft_consensus.cc:3058] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:57.677717 28153 raft_consensus.cc:2466] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:57.698549 28595 raft_consensus.cc:491] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:58:57.699066 28595 raft_consensus.cc:513] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.701807 28595 leader_election.cc:290] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:57.703155 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:58:57.703817 28006 raft_consensus.cc:2466] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:57.704231 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:58:57.704885 28153 raft_consensus.cc:2466] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 0.
I20250905 19:58:57.705760 28050 leader_election.cc:304] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:57.706688 28589 raft_consensus.cc:2802] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:58:57.707058 28589 raft_consensus.cc:491] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:58:57.707386 28589 raft_consensus.cc:3058] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:57.714780 28589 raft_consensus.cc:513] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.716673 28589 leader_election.cc:290] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:58:57.716164 27905 catalog_manager.cc:5582] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:57.719810 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:58:57.720431 28006 raft_consensus.cc:3058] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:57.720589 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:58:57.721195 28153 raft_consensus.cc:3058] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:57.726969 28006 raft_consensus.cc:2466] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:57.728132 28051 leader_election.cc:304] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:58:57.728936 28589 raft_consensus.cc:2802] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:58:57.729416 28589 raft_consensus.cc:695] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:58:57.730147 28153 raft_consensus.cc:2466] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:58:57.730388 28589 consensus_queue.cc:237] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:58:57.741402 27907 catalog_manager.cc:5582] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:58:57.770771 28001 raft_consensus.cc:3058] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:58:57.875756 28589 consensus_queue.cc:1035] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:57.893688 28589 consensus_queue.cc:1035] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:58.026829 28595 consensus_queue.cc:1035] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:58.040344 28595 consensus_queue.cc:1035] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:58.082022 28616 consensus_queue.cc:1035] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:58.088472 28589 consensus_queue.cc:1035] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:58.095042 28616 consensus_queue.cc:1035] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:58.116580 28595 consensus_queue.cc:1035] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:58.197805 28589 consensus_queue.cc:1035] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:58:58.213584 28593 consensus_queue.cc:1035] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:58:59.066882 26858 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:58:59.104748 28709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:58:59.104779 28706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:58:59.104748 28707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:58:59.105561 26858 server_base.cc:1047] running on GCE node
I20250905 19:58:59.106959 26858 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 19:58:59.107197 26858 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 19:58:59.107385 26858 hybrid_clock.cc:648] HybridClock initialized: now 1757102339107366 us; error 0 us; skew 500 ppm
I20250905 19:58:59.108289 26858 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:58:59.111436 26858 webserver.cc:480] Webserver started at http://127.26.58.132:39477/ using document root <none> and password file <none>
I20250905 19:58:59.112053 26858 fs_manager.cc:362] Metadata directory not provided
I20250905 19:58:59.112282 26858 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:58:59.112653 26858 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:58:59.114660 26858 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-3-root/instance:
uuid: "7e9c9f591f05464f9b097bf003d14f4a"
format_stamp: "Formatted at 2025-09-05 19:58:59 on dist-test-slave-v26w"
I20250905 19:58:59.121140 26858 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20250905 19:58:59.125020 28714 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:58:59.125902 26858 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 19:58:59.126232 26858 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-3-root
uuid: "7e9c9f591f05464f9b097bf003d14f4a"
format_stamp: "Formatted at 2025-09-05 19:58:59 on dist-test-slave-v26w"
I20250905 19:58:59.126591 26858 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:58:59.149629 26858 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:58:59.151047 26858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:58:59.152392 26858 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:58:59.155321 26858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:58:59.155593 26858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:58:59.155918 26858 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:58:59.156104 26858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:58:59.222994 26858 rpc_server.cc:307] RPC server started. Bound to: 127.26.58.132:44299
I20250905 19:58:59.223110 28776 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.58.132:44299 every 8 connection(s)
I20250905 19:58:59.236346 26858 auto_leader_rebalancer-test.cc:417] add a tserver: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:58:59.251258 28777 heartbeater.cc:344] Connected to a master server at 127.26.58.190:38969
I20250905 19:58:59.251780 28777 heartbeater.cc:461] Registering TS with master...
I20250905 19:58:59.252820 28777 heartbeater.cc:507] Master 127.26.58.190:38969 requested a full tablet report, sending...
I20250905 19:58:59.255230 27907 ts_manager.cc:194] Registered new tserver with Master: 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:58:59.256796 27907 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60818
I20250905 19:58:59.610157 28001 consensus_queue.cc:237] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:58:59.618850 28081 raft_consensus.cc:1273] T 8afd77a8d80843aa805f971a2806ab86 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:58:59.621598 28705 consensus_queue.cc:1035] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:58:59.625448 28153 raft_consensus.cc:1273] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:58:59.627400 28705 consensus_queue.cc:1035] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:58:59.675885 28586 raft_consensus.cc:2953] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
W20250905 19:58:59.684450 27974 consensus_peers.cc:489] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 -> Peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299): Couldn't send request to peer 7e9c9f591f05464f9b097bf003d14f4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8afd77a8d80843aa805f971a2806ab86. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:58:59.685207 28154 raft_consensus.cc:2953] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:58:59.693662 28081 raft_consensus.cc:2953] T 8afd77a8d80843aa805f971a2806ab86 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:58:59.695446 28002 consensus_queue.cc:237] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:58:59.700429 27907 catalog_manager.cc:5582] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250905 19:58:59.718988 27974 consensus_peers.cc:489] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 -> Peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299): Couldn't send request to peer 7e9c9f591f05464f9b097bf003d14f4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2bd6bb972f14472a89ccf99de880efc2. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:58:59.721841 28154 raft_consensus.cc:1273] T 2bd6bb972f14472a89ccf99de880efc2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:58:59.724848 28586 consensus_queue.cc:1035] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:58:59.726219 28081 raft_consensus.cc:1273] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:58:59.728003 28586 consensus_queue.cc:1035] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:58:59.740808 28793 raft_consensus.cc:2953] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:58:59.742789 28154 raft_consensus.cc:2953] T 2bd6bb972f14472a89ccf99de880efc2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:58:59.769946 28081 raft_consensus.cc:2953] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:58:59.775969 27905 catalog_manager.cc:5582] T 2bd6bb972f14472a89ccf99de880efc2 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:58:59.792732 28081 consensus_queue.cc:237] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:58:59.800526 28153 raft_consensus.cc:1273] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:58:59.800817 28002 raft_consensus.cc:1273] T 4f2279535926437ba84ae91f2b83b3d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:58:59.802826 28786 consensus_queue.cc:1035] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:58:59.808141 28788 consensus_queue.cc:1035] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:58:59.827255 28593 raft_consensus.cc:2953] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:58:59.831271 28153 raft_consensus.cc:2953] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:58:59.846319 28002 raft_consensus.cc:2953] T 4f2279535926437ba84ae91f2b83b3d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:58:59.855226 27908 catalog_manager.cc:5582] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250905 19:58:59.876430 28049 consensus_peers.cc:489] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 -> Peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299): Couldn't send request to peer 7e9c9f591f05464f9b097bf003d14f4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f2279535926437ba84ae91f2b83b3d3. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:58:59.885412 28153 consensus_queue.cc:237] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:58:59.894779 28081 raft_consensus.cc:1273] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:58:59.896155 28002 raft_consensus.cc:1273] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:58:59.896463 28488 consensus_queue.cc:1035] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:58:59.897855 28629 consensus_queue.cc:1035] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:58:59.910032 28629 raft_consensus.cc:2953] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:58:59.915601 28002 raft_consensus.cc:2953] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
W20250905 19:58:59.924948 28123 consensus_peers.cc:489] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 -> Peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299): Couldn't send request to peer 7e9c9f591f05464f9b097bf003d14f4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a9976ebf1c1415a85592e195bad9f36. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:58:59.940026 28083 raft_consensus.cc:2953] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:58:59.943689 27907 catalog_manager.cc:5582] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:00.001724 28812 ts_tablet_manager.cc:927] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a: Initiating tablet copy from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421)
I20250905 19:59:00.004884 28812 tablet_copy_client.cc:323] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Beginning tablet copy session from remote peer at address 127.26.58.129:38421
I20250905 19:59:00.023164 28020 tablet_copy_service.cc:140] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Received BeginTabletCopySession request for tablet 8afd77a8d80843aa805f971a2806ab86 from peer 7e9c9f591f05464f9b097bf003d14f4a ({username='slave'} at 127.0.0.1:45844)
I20250905 19:59:00.023948 28020 tablet_copy_service.cc:161] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Beginning new tablet copy session on tablet 8afd77a8d80843aa805f971a2806ab86 from peer 7e9c9f591f05464f9b097bf003d14f4a at {username='slave'} at 127.0.0.1:45844: session id = 7e9c9f591f05464f9b097bf003d14f4a-8afd77a8d80843aa805f971a2806ab86
I20250905 19:59:00.032999 28020 tablet_copy_source_session.cc:215] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:00.036810 28812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8afd77a8d80843aa805f971a2806ab86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:00.055536 28812 tablet_copy_client.cc:806] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:00.056339 28812 tablet_copy_client.cc:670] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:00.062552 28812 tablet_copy_client.cc:538] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 19:59:00.076414 28812 tablet_bootstrap.cc:492] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap starting.
I20250905 19:59:00.098489 28812 tablet_bootstrap.cc:492] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:00.099422 28812 tablet_bootstrap.cc:492] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap complete.
I20250905 19:59:00.100143 28812 ts_tablet_manager.cc:1397] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent bootstrapping tablet: real 0.024s user 0.019s sys 0.005s
I20250905 19:59:00.103407 28812 raft_consensus.cc:357] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:00.105877 28812 raft_consensus.cc:738] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Initialized, Role: LEARNER
I20250905 19:59:00.106511 28812 consensus_queue.cc:260] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:00.109431 28777 heartbeater.cc:499] Master 127.26.58.190:38969 was elected leader, sending a full tablet report...
I20250905 19:59:00.110049 28812 ts_tablet_manager.cc:1428] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20250905 19:59:00.111775 28020 tablet_copy_service.cc:342] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Request end of tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-8afd77a8d80843aa805f971a2806ab86 received from {username='slave'} at 127.0.0.1:45844
I20250905 19:59:00.112278 28020 tablet_copy_service.cc:434] P cd764bfeebac4cc9ad851cd4a6a7d3b0: ending tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-8afd77a8d80843aa805f971a2806ab86 on tablet 8afd77a8d80843aa805f971a2806ab86 with peer 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:00.220989 28812 ts_tablet_manager.cc:927] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a: Initiating tablet copy from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421)
I20250905 19:59:00.222376 28812 tablet_copy_client.cc:323] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Beginning tablet copy session from remote peer at address 127.26.58.129:38421
I20250905 19:59:00.223930 28020 tablet_copy_service.cc:140] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Received BeginTabletCopySession request for tablet 2bd6bb972f14472a89ccf99de880efc2 from peer 7e9c9f591f05464f9b097bf003d14f4a ({username='slave'} at 127.0.0.1:45844)
I20250905 19:59:00.224445 28020 tablet_copy_service.cc:161] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Beginning new tablet copy session on tablet 2bd6bb972f14472a89ccf99de880efc2 from peer 7e9c9f591f05464f9b097bf003d14f4a at {username='slave'} at 127.0.0.1:45844: session id = 7e9c9f591f05464f9b097bf003d14f4a-2bd6bb972f14472a89ccf99de880efc2
I20250905 19:59:00.230773 28020 tablet_copy_source_session.cc:215] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:00.233894 28812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bd6bb972f14472a89ccf99de880efc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:00.249116 28812 tablet_copy_client.cc:806] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:00.249846 28812 tablet_copy_client.cc:670] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:00.253407 28812 tablet_copy_client.cc:538] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 19:59:00.261232 28812 tablet_bootstrap.cc:492] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap starting.
I20250905 19:59:00.283278 28812 tablet_bootstrap.cc:492] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:00.284279 28812 tablet_bootstrap.cc:492] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap complete.
I20250905 19:59:00.284929 28812 ts_tablet_manager.cc:1397] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent bootstrapping tablet: real 0.024s user 0.022s sys 0.000s
I20250905 19:59:00.287986 28812 raft_consensus.cc:357] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:00.288832 28812 raft_consensus.cc:738] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Initialized, Role: LEARNER
I20250905 19:59:00.289448 28812 consensus_queue.cc:260] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:00.291747 28812 ts_tablet_manager.cc:1428] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20250905 19:59:00.296121 28020 tablet_copy_service.cc:342] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Request end of tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-2bd6bb972f14472a89ccf99de880efc2 received from {username='slave'} at 127.0.0.1:45844
I20250905 19:59:00.296597 28020 tablet_copy_service.cc:434] P cd764bfeebac4cc9ad851cd4a6a7d3b0: ending tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-2bd6bb972f14472a89ccf99de880efc2 on tablet 2bd6bb972f14472a89ccf99de880efc2 with peer 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:00.309471 28812 ts_tablet_manager.cc:927] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a: Initiating tablet copy from peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:00.311174 28812 tablet_copy_client.cc:323] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Beginning tablet copy session from remote peer at address 127.26.58.130:34193
I20250905 19:59:00.325672 28095 tablet_copy_service.cc:140] P e4413ae5e0b64b52a6f360ac580239b9: Received BeginTabletCopySession request for tablet 4f2279535926437ba84ae91f2b83b3d3 from peer 7e9c9f591f05464f9b097bf003d14f4a ({username='slave'} at 127.0.0.1:50660)
I20250905 19:59:00.326175 28095 tablet_copy_service.cc:161] P e4413ae5e0b64b52a6f360ac580239b9: Beginning new tablet copy session on tablet 4f2279535926437ba84ae91f2b83b3d3 from peer 7e9c9f591f05464f9b097bf003d14f4a at {username='slave'} at 127.0.0.1:50660: session id = 7e9c9f591f05464f9b097bf003d14f4a-4f2279535926437ba84ae91f2b83b3d3
I20250905 19:59:00.337555 28095 tablet_copy_source_session.cc:215] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:00.340847 28812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f2279535926437ba84ae91f2b83b3d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:00.355393 28822 ts_tablet_manager.cc:927] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a: Initiating tablet copy from peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:00.357213 28812 tablet_copy_client.cc:806] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:00.357882 28812 tablet_copy_client.cc:670] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:00.360416 28822 tablet_copy_client.cc:323] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Beginning tablet copy session from remote peer at address 127.26.58.131:36705
I20250905 19:59:00.366938 28812 tablet_copy_client.cc:538] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 19:59:00.376070 28812 tablet_bootstrap.cc:492] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap starting.
I20250905 19:59:00.386215 28169 tablet_copy_service.cc:140] P 26b2f874b8e54f08b01a9ed00d5cce05: Received BeginTabletCopySession request for tablet 9a9976ebf1c1415a85592e195bad9f36 from peer 7e9c9f591f05464f9b097bf003d14f4a ({username='slave'} at 127.0.0.1:39586)
I20250905 19:59:00.386914 28169 tablet_copy_service.cc:161] P 26b2f874b8e54f08b01a9ed00d5cce05: Beginning new tablet copy session on tablet 9a9976ebf1c1415a85592e195bad9f36 from peer 7e9c9f591f05464f9b097bf003d14f4a at {username='slave'} at 127.0.0.1:39586: session id = 7e9c9f591f05464f9b097bf003d14f4a-9a9976ebf1c1415a85592e195bad9f36
I20250905 19:59:00.400357 28812 tablet_bootstrap.cc:492] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:00.401166 28812 tablet_bootstrap.cc:492] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap complete.
I20250905 19:59:00.401525 28169 tablet_copy_source_session.cc:215] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:00.401690 28812 ts_tablet_manager.cc:1397] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.011s
I20250905 19:59:00.404176 28812 raft_consensus.cc:357] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:00.405148 28812 raft_consensus.cc:738] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Initialized, Role: LEARNER
I20250905 19:59:00.405828 28812 consensus_queue.cc:260] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:00.406533 28822 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a9976ebf1c1415a85592e195bad9f36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:00.409540 28812 ts_tablet_manager.cc:1428] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20250905 19:59:00.413813 28095 tablet_copy_service.cc:342] P e4413ae5e0b64b52a6f360ac580239b9: Request end of tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-4f2279535926437ba84ae91f2b83b3d3 received from {username='slave'} at 127.0.0.1:50660
I20250905 19:59:00.414327 28095 tablet_copy_service.cc:434] P e4413ae5e0b64b52a6f360ac580239b9: ending tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-4f2279535926437ba84ae91f2b83b3d3 on tablet 4f2279535926437ba84ae91f2b83b3d3 with peer 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:00.428608 28822 tablet_copy_client.cc:806] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:00.429275 28822 tablet_copy_client.cc:670] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:00.433048 28822 tablet_copy_client.cc:538] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 19:59:00.443192 28822 tablet_bootstrap.cc:492] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap starting.
I20250905 19:59:00.463936 28822 tablet_bootstrap.cc:492] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:00.464830 28822 tablet_bootstrap.cc:492] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap complete.
I20250905 19:59:00.465490 28822 ts_tablet_manager.cc:1397] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent bootstrapping tablet: real 0.023s user 0.015s sys 0.004s
I20250905 19:59:00.468043 28822 raft_consensus.cc:357] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:00.468879 28822 raft_consensus.cc:738] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Initialized, Role: LEARNER
I20250905 19:59:00.469493 28822 consensus_queue.cc:260] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:00.472044 28822 ts_tablet_manager.cc:1428] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20250905 19:59:00.474032 28169 tablet_copy_service.cc:342] P 26b2f874b8e54f08b01a9ed00d5cce05: Request end of tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-9a9976ebf1c1415a85592e195bad9f36 received from {username='slave'} at 127.0.0.1:39586
I20250905 19:59:00.474545 28169 tablet_copy_service.cc:434] P 26b2f874b8e54f08b01a9ed00d5cce05: ending tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-9a9976ebf1c1415a85592e195bad9f36 on tablet 9a9976ebf1c1415a85592e195bad9f36 with peer 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:00.594718 28752 raft_consensus.cc:1215] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250905 19:59:00.685288 28752 raft_consensus.cc:1215] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250905 19:59:00.820814 28752 raft_consensus.cc:1215] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250905 19:59:00.899977 28752 raft_consensus.cc:1215] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250905 19:59:01.092399 28821 raft_consensus.cc:1062] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0: attempting to promote NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a to VOTER
I20250905 19:59:01.095551 28821 consensus_queue.cc:237] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:01.102905 28752 raft_consensus.cc:1273] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:01.105437 28586 consensus_queue.cc:1035] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:01.106245 28082 raft_consensus.cc:1273] T 8afd77a8d80843aa805f971a2806ab86 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:01.106802 28153 raft_consensus.cc:1273] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:01.107923 28586 consensus_queue.cc:1035] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:01.109558 28821 consensus_queue.cc:1035] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:01.117399 28586 raft_consensus.cc:2953] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.124989 28082 raft_consensus.cc:2953] T 8afd77a8d80843aa805f971a2806ab86 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.127224 28752 raft_consensus.cc:2953] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.127280 28153 raft_consensus.cc:2953] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.142964 27907 catalog_manager.cc:5582] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 19:59:01.175091 28006 consensus_queue.cc:237] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:01.181743 28752 raft_consensus.cc:1273] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:01.191257 28821 consensus_queue.cc:1035] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 19:59:01.199636 28586 raft_consensus.cc:2953] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.211637 28752 raft_consensus.cc:2953] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.219368 27895 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 8afd77a8d80843aa805f971a2806ab86 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 19:59:01.220628 28153 raft_consensus.cc:1273] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:01.222587 28828 consensus_queue.cc:1035] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:59:01.222985 28798 raft_consensus.cc:1062] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: attempting to promote NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a to VOTER
I20250905 19:59:01.225587 28798 consensus_queue.cc:237] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:01.228818 28153 raft_consensus.cc:2953] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.227963 27905 catalog_manager.cc:5582] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 19:59:01.236464 28752 raft_consensus.cc:1273] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:01.236481 28154 raft_consensus.cc:1273] T 2bd6bb972f14472a89ccf99de880efc2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:01.238152 28586 consensus_queue.cc:1035] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:01.244033 28586 consensus_queue.cc:1035] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:01.246681 28072 tablet_service.cc:1515] Processing DeleteTablet for tablet 8afd77a8d80843aa805f971a2806ab86 with delete_type TABLET_DATA_TOMBSTONED (TS e4413ae5e0b64b52a6f360ac580239b9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59536
I20250905 19:59:01.248237 28082 raft_consensus.cc:1273] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:01.249748 28821 consensus_queue.cc:1035] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20250905 19:59:01.278053 28798 raft_consensus.cc:2953] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.284868 28786 raft_consensus.cc:1062] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9: attempting to promote NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a to VOTER
I20250905 19:59:01.286787 28083 raft_consensus.cc:2953] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.293967 28786 consensus_queue.cc:237] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:01.295060 28154 raft_consensus.cc:2953] T 2bd6bb972f14472a89ccf99de880efc2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.283548 28838 tablet_replica.cc:331] T 8afd77a8d80843aa805f971a2806ab86 P e4413ae5e0b64b52a6f360ac580239b9: stopping tablet replica
I20250905 19:59:01.298214 28838 raft_consensus.cc:2241] T 8afd77a8d80843aa805f971a2806ab86 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:01.299135 28838 raft_consensus.cc:2270] T 8afd77a8d80843aa805f971a2806ab86 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:01.304524 28752 raft_consensus.cc:2953] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.314831 28838 ts_tablet_manager.cc:1905] T 8afd77a8d80843aa805f971a2806ab86 P e4413ae5e0b64b52a6f360ac580239b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 19:59:01.322270 27908 catalog_manager.cc:5582] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 19:59:01.328390 28826 raft_consensus.cc:1062] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05: attempting to promote NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a to VOTER
I20250905 19:59:01.338405 28002 raft_consensus.cc:1273] T 4f2279535926437ba84ae91f2b83b3d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:01.340166 28751 raft_consensus.cc:1273] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:01.340333 28843 consensus_queue.cc:1035] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:01.342098 28154 raft_consensus.cc:1273] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:01.343128 28788 consensus_queue.cc:1035] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:01.346019 28006 consensus_queue.cc:237] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:01.349628 28826 consensus_queue.cc:237] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:01.352306 28593 consensus_queue.cc:1035] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:01.378417 28752 raft_consensus.cc:1273] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:01.380164 28828 consensus_queue.cc:1035] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:59:01.388630 28752 raft_consensus.cc:1273] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:01.389285 28828 raft_consensus.cc:2953] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.391417 28080 raft_consensus.cc:1273] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:01.392728 28810 consensus_queue.cc:1035] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:01.393452 28752 raft_consensus.cc:2953] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.398643 28080 raft_consensus.cc:1273] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:01.395463 28798 consensus_queue.cc:1035] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20250905 19:59:01.401320 28006 raft_consensus.cc:1273] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:01.402612 28810 consensus_queue.cc:1035] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:01.408917 28080 raft_consensus.cc:2953] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.414677 28838 ts_tablet_manager.cc:1918] T 8afd77a8d80843aa805f971a2806ab86 P e4413ae5e0b64b52a6f360ac580239b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250905 19:59:01.415341 28838 log.cc:1199] T 8afd77a8d80843aa805f971a2806ab86 P e4413ae5e0b64b52a6f360ac580239b9: Deleting WAL directory at /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-1-root/wals/8afd77a8d80843aa805f971a2806ab86
I20250905 19:59:01.417837 27892 catalog_manager.cc:4928] TS e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): tablet 8afd77a8d80843aa805f971a2806ab86 (table test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]) successfully deleted
I20250905 19:59:01.420594 28810 consensus_queue.cc:1035] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:01.425709 27908 catalog_manager.cc:5582] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.436761 28145 tablet_service.cc:1515] Processing DeleteTablet for tablet 2bd6bb972f14472a89ccf99de880efc2 with delete_type TABLET_DATA_TOMBSTONED (TS 26b2f874b8e54f08b01a9ed00d5cce05 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33742
I20250905 19:59:01.439522 28851 tablet_replica.cc:331] T 2bd6bb972f14472a89ccf99de880efc2 P 26b2f874b8e54f08b01a9ed00d5cce05: stopping tablet replica
I20250905 19:59:01.440692 28851 raft_consensus.cc:2241] T 2bd6bb972f14472a89ccf99de880efc2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:01.441571 28851 raft_consensus.cc:2270] T 2bd6bb972f14472a89ccf99de880efc2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:01.447261 28851 ts_tablet_manager.cc:1905] T 2bd6bb972f14472a89ccf99de880efc2 P 26b2f874b8e54f08b01a9ed00d5cce05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 19:59:01.457765 28810 raft_consensus.cc:2953] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.458611 28786 raft_consensus.cc:2953] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.460753 27895 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 2bd6bb972f14472a89ccf99de880efc2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 19:59:01.465759 28002 raft_consensus.cc:2953] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.476608 28751 raft_consensus.cc:2953] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.472532 28154 raft_consensus.cc:2953] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.494912 27908 catalog_manager.cc:5582] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.475152 28083 raft_consensus.cc:2953] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.509789 28751 raft_consensus.cc:2953] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.523618 28851 ts_tablet_manager.cc:1918] T 2bd6bb972f14472a89ccf99de880efc2 P 26b2f874b8e54f08b01a9ed00d5cce05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250905 19:59:01.524199 28851 log.cc:1199] T 2bd6bb972f14472a89ccf99de880efc2 P 26b2f874b8e54f08b01a9ed00d5cce05: Deleting WAL directory at /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-2-root/wals/2bd6bb972f14472a89ccf99de880efc2
I20250905 19:59:01.555060 27908 catalog_manager.cc:5582] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.556874 28006 raft_consensus.cc:2953] T 4f2279535926437ba84ae91f2b83b3d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.554534 28083 consensus_queue.cc:237] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:01.568892 27894 catalog_manager.cc:4928] TS 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): tablet 2bd6bb972f14472a89ccf99de880efc2 (table test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]) successfully deleted
I20250905 19:59:01.587020 28751 raft_consensus.cc:1273] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:01.588567 28788 consensus_queue.cc:1035] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:59:01.590622 28154 consensus_queue.cc:237] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:01.596913 28750 raft_consensus.cc:1273] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:01.598634 28854 consensus_queue.cc:1035] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 19:59:01.599573 28593 raft_consensus.cc:2953] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.601945 28154 raft_consensus.cc:1273] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:01.603183 28750 raft_consensus.cc:2953] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.606696 28843 consensus_queue.cc:1035] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:59:01.610666 28156 raft_consensus.cc:2953] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.613301 28853 raft_consensus.cc:2953] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.620550 28082 raft_consensus.cc:1273] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:01.622033 28854 consensus_queue.cc:1035] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:59:01.619619 28752 raft_consensus.cc:2953] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.634200 27892 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 4f2279535926437ba84ae91f2b83b3d3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 19:59:01.633423 28082 raft_consensus.cc:2953] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.645824 27907 catalog_manager.cc:5582] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.646907 27894 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9a9976ebf1c1415a85592e195bad9f36 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 19:59:01.657277 28000 tablet_service.cc:1515] Processing DeleteTablet for tablet 4f2279535926437ba84ae91f2b83b3d3 with delete_type TABLET_DATA_TOMBSTONED (TS cd764bfeebac4cc9ad851cd4a6a7d3b0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54144
I20250905 19:59:01.664595 28856 tablet_replica.cc:331] T 4f2279535926437ba84ae91f2b83b3d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: stopping tablet replica
I20250905 19:59:01.665750 28856 raft_consensus.cc:2241] T 4f2279535926437ba84ae91f2b83b3d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:01.665108 27907 catalog_manager.cc:5582] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:01.666541 28856 raft_consensus.cc:2270] T 4f2279535926437ba84ae91f2b83b3d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:01.670305 28856 ts_tablet_manager.cc:1905] T 4f2279535926437ba84ae91f2b83b3d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 19:59:01.680833 28000 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a9976ebf1c1415a85592e195bad9f36 with delete_type TABLET_DATA_TOMBSTONED (TS cd764bfeebac4cc9ad851cd4a6a7d3b0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54144
I20250905 19:59:01.693322 28856 ts_tablet_manager.cc:1918] T 4f2279535926437ba84ae91f2b83b3d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250905 19:59:01.693872 28856 log.cc:1199] T 4f2279535926437ba84ae91f2b83b3d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Deleting WAL directory at /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-0-root/wals/4f2279535926437ba84ae91f2b83b3d3
I20250905 19:59:01.698354 28856 tablet_replica.cc:331] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0: stopping tablet replica
I20250905 19:59:01.698689 27895 catalog_manager.cc:4928] TS cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): tablet 4f2279535926437ba84ae91f2b83b3d3 (table test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]) successfully deleted
I20250905 19:59:01.699514 28856 raft_consensus.cc:2241] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:01.700273 28856 raft_consensus.cc:2270] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:01.703719 28856 ts_tablet_manager.cc:1905] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 19:59:01.716885 28856 ts_tablet_manager.cc:1918] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250905 19:59:01.717231 28856 log.cc:1199] T 9a9976ebf1c1415a85592e195bad9f36 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Deleting WAL directory at /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-0-root/wals/9a9976ebf1c1415a85592e195bad9f36
I20250905 19:59:01.718564 27895 catalog_manager.cc:4928] TS cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): tablet 9a9976ebf1c1415a85592e195bad9f36 (table test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]) successfully deleted
I20250905 19:59:02.720863 28083 consensus_queue.cc:237] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:02.727540 28154 raft_consensus.cc:1273] T 361ffd5c9a994e0397e6748827466d3d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250905 19:59:02.728389 28049 consensus_peers.cc:489] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 -> Peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299): Couldn't send request to peer 7e9c9f591f05464f9b097bf003d14f4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 361ffd5c9a994e0397e6748827466d3d. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:02.728760 28004 raft_consensus.cc:1273] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:02.730371 28858 consensus_queue.cc:1035] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:02.731364 28593 consensus_queue.cc:1035] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:02.740240 28593 raft_consensus.cc:2953] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:02.741714 28154 raft_consensus.cc:2953] T 361ffd5c9a994e0397e6748827466d3d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:02.744060 28006 raft_consensus.cc:2953] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:02.759963 28006 consensus_queue.cc:237] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:02.760066 27908 catalog_manager.cc:5582] T 361ffd5c9a994e0397e6748827466d3d P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:02.772886 28156 raft_consensus.cc:1273] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:02.774314 28828 consensus_queue.cc:1035] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250905 19:59:02.775420 27974 consensus_peers.cc:489] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 -> Peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299): Couldn't send request to peer 7e9c9f591f05464f9b097bf003d14f4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9f7f6e4e3a754dcda90928621168fbe7. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:02.778092 28083 raft_consensus.cc:1273] T 9f7f6e4e3a754dcda90928621168fbe7 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:02.784189 28828 consensus_queue.cc:1035] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20250905 19:59:02.808624 28857 raft_consensus.cc:2953] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:02.810124 28156 raft_consensus.cc:2953] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:02.818486 28082 raft_consensus.cc:2953] T 9f7f6e4e3a754dcda90928621168fbe7 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:02.827495 28156 consensus_queue.cc:237] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
W20250905 19:59:02.839946 28123 consensus_peers.cc:489] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 -> Peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299): Couldn't send request to peer 7e9c9f591f05464f9b097bf003d14f4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a8d8393352b4214a20c1683852d4fdd. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:02.841536 28082 raft_consensus.cc:1273] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:02.842171 28006 raft_consensus.cc:1273] T 8a8d8393352b4214a20c1683852d4fdd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:02.842932 28810 consensus_queue.cc:1035] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:02.843639 27907 catalog_manager.cc:5582] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:02.844509 28810 consensus_queue.cc:1035] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:02.875187 28855 raft_consensus.cc:2953] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:02.883275 28006 raft_consensus.cc:2953] T 8a8d8393352b4214a20c1683852d4fdd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:02.892211 28002 consensus_queue.cc:237] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:02.897258 27908 catalog_manager.cc:5582] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:02.903141 28080 raft_consensus.cc:2953] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:02.909298 28156 raft_consensus.cc:1273] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20250905 19:59:02.909885 27974 consensus_peers.cc:489] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 -> Peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299): Couldn't send request to peer 7e9c9f591f05464f9b097bf003d14f4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc6a8d5d253643cea8582bef1c4173d6. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:02.911181 28857 consensus_queue.cc:1035] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:02.922906 28082 raft_consensus.cc:1273] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:59:02.924490 28869 raft_consensus.cc:2953] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:02.925959 28586 consensus_queue.cc:1035] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:02.933492 28156 raft_consensus.cc:2953] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:02.938061 28080 raft_consensus.cc:2953] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:02.950214 27905 catalog_manager.cc:5582] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index -1 to 3, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New cstate: current_term: 3 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:03.292832 28878 ts_tablet_manager.cc:927] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a: Initiating tablet copy from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421)
I20250905 19:59:03.295969 28879 ts_tablet_manager.cc:927] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a: Initiating tablet copy from peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:03.297366 28878 tablet_copy_client.cc:323] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Beginning tablet copy session from remote peer at address 127.26.58.129:38421
I20250905 19:59:03.299307 28020 tablet_copy_service.cc:140] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Received BeginTabletCopySession request for tablet 9f7f6e4e3a754dcda90928621168fbe7 from peer 7e9c9f591f05464f9b097bf003d14f4a ({username='slave'} at 127.0.0.1:45844)
I20250905 19:59:03.299767 28879 tablet_copy_client.cc:323] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Beginning tablet copy session from remote peer at address 127.26.58.131:36705
I20250905 19:59:03.299927 28020 tablet_copy_service.cc:161] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Beginning new tablet copy session on tablet 9f7f6e4e3a754dcda90928621168fbe7 from peer 7e9c9f591f05464f9b097bf003d14f4a at {username='slave'} at 127.0.0.1:45844: session id = 7e9c9f591f05464f9b097bf003d14f4a-9f7f6e4e3a754dcda90928621168fbe7
I20250905 19:59:03.301522 28169 tablet_copy_service.cc:140] P 26b2f874b8e54f08b01a9ed00d5cce05: Received BeginTabletCopySession request for tablet 8a8d8393352b4214a20c1683852d4fdd from peer 7e9c9f591f05464f9b097bf003d14f4a ({username='slave'} at 127.0.0.1:39586)
I20250905 19:59:03.302048 28169 tablet_copy_service.cc:161] P 26b2f874b8e54f08b01a9ed00d5cce05: Beginning new tablet copy session on tablet 8a8d8393352b4214a20c1683852d4fdd from peer 7e9c9f591f05464f9b097bf003d14f4a at {username='slave'} at 127.0.0.1:39586: session id = 7e9c9f591f05464f9b097bf003d14f4a-8a8d8393352b4214a20c1683852d4fdd
I20250905 19:59:03.307691 28020 tablet_copy_source_session.cc:215] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:03.311195 28878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f7f6e4e3a754dcda90928621168fbe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:03.311309 28169 tablet_copy_source_session.cc:215] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:03.314482 28882 ts_tablet_manager.cc:927] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a: Initiating tablet copy from peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:03.319340 28882 tablet_copy_client.cc:323] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Beginning tablet copy session from remote peer at address 127.26.58.130:34193
I20250905 19:59:03.322952 28879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a8d8393352b4214a20c1683852d4fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:03.323647 28095 tablet_copy_service.cc:140] P e4413ae5e0b64b52a6f360ac580239b9: Received BeginTabletCopySession request for tablet 361ffd5c9a994e0397e6748827466d3d from peer 7e9c9f591f05464f9b097bf003d14f4a ({username='slave'} at 127.0.0.1:50660)
I20250905 19:59:03.324235 28095 tablet_copy_service.cc:161] P e4413ae5e0b64b52a6f360ac580239b9: Beginning new tablet copy session on tablet 361ffd5c9a994e0397e6748827466d3d from peer 7e9c9f591f05464f9b097bf003d14f4a at {username='slave'} at 127.0.0.1:50660: session id = 7e9c9f591f05464f9b097bf003d14f4a-361ffd5c9a994e0397e6748827466d3d
I20250905 19:59:03.330892 28878 tablet_copy_client.cc:806] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:03.331605 28878 tablet_copy_client.cc:670] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:03.337765 28095 tablet_copy_source_session.cc:215] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:03.338531 28878 tablet_copy_client.cc:538] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 19:59:03.341513 28882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 361ffd5c9a994e0397e6748827466d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:03.347810 28878 tablet_bootstrap.cc:492] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap starting.
I20250905 19:59:03.355705 28879 tablet_copy_client.cc:806] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:03.356375 28879 tablet_copy_client.cc:670] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:03.369298 28882 tablet_copy_client.cc:806] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:03.369803 28879 tablet_copy_client.cc:538] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 19:59:03.379207 28882 tablet_copy_client.cc:670] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:03.380900 28878 tablet_bootstrap.cc:492] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:03.380576 28879 tablet_bootstrap.cc:492] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap starting.
I20250905 19:59:03.382102 28878 tablet_bootstrap.cc:492] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap complete.
I20250905 19:59:03.389130 28878 ts_tablet_manager.cc:1397] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent bootstrapping tablet: real 0.042s user 0.015s sys 0.013s
I20250905 19:59:03.390563 28882 tablet_copy_client.cc:538] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 19:59:03.393383 28878 raft_consensus.cc:357] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:03.394459 28878 raft_consensus.cc:738] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Initialized, Role: LEARNER
I20250905 19:59:03.395211 28878 consensus_queue.cc:260] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:03.399582 28882 tablet_bootstrap.cc:492] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap starting.
I20250905 19:59:03.403635 28878 ts_tablet_manager.cc:1428] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20250905 19:59:03.408527 28020 tablet_copy_service.cc:342] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Request end of tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-9f7f6e4e3a754dcda90928621168fbe7 received from {username='slave'} at 127.0.0.1:45844
I20250905 19:59:03.409003 28020 tablet_copy_service.cc:434] P cd764bfeebac4cc9ad851cd4a6a7d3b0: ending tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-9f7f6e4e3a754dcda90928621168fbe7 on tablet 9f7f6e4e3a754dcda90928621168fbe7 with peer 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:03.443135 28879 tablet_bootstrap.cc:492] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:03.444444 28879 tablet_bootstrap.cc:492] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap complete.
I20250905 19:59:03.445307 28879 ts_tablet_manager.cc:1397] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a: Time spent bootstrapping tablet: real 0.065s user 0.010s sys 0.020s
I20250905 19:59:03.448622 28879 raft_consensus.cc:357] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:03.449236 28882 tablet_bootstrap.cc:492] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:03.449647 28879 raft_consensus.cc:738] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Initialized, Role: LEARNER
I20250905 19:59:03.450383 28882 tablet_bootstrap.cc:492] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap complete.
I20250905 19:59:03.450491 28879 consensus_queue.cc:260] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:03.451225 28882 ts_tablet_manager.cc:1397] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a: Time spent bootstrapping tablet: real 0.052s user 0.015s sys 0.011s
I20250905 19:59:03.453212 28879 ts_tablet_manager.cc:1428] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250905 19:59:03.455124 28882 raft_consensus.cc:357] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:03.456036 28882 raft_consensus.cc:738] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Initialized, Role: LEARNER
I20250905 19:59:03.456573 28169 tablet_copy_service.cc:342] P 26b2f874b8e54f08b01a9ed00d5cce05: Request end of tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-8a8d8393352b4214a20c1683852d4fdd received from {username='slave'} at 127.0.0.1:39586
I20250905 19:59:03.457041 28169 tablet_copy_service.cc:434] P 26b2f874b8e54f08b01a9ed00d5cce05: ending tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-8a8d8393352b4214a20c1683852d4fdd on tablet 8a8d8393352b4214a20c1683852d4fdd with peer 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:03.456724 28882 consensus_queue.cc:260] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:03.460485 28882 ts_tablet_manager.cc:1428] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250905 19:59:03.462116 28095 tablet_copy_service.cc:342] P e4413ae5e0b64b52a6f360ac580239b9: Request end of tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-361ffd5c9a994e0397e6748827466d3d received from {username='slave'} at 127.0.0.1:50660
I20250905 19:59:03.462525 28095 tablet_copy_service.cc:434] P e4413ae5e0b64b52a6f360ac580239b9: ending tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-361ffd5c9a994e0397e6748827466d3d on tablet 361ffd5c9a994e0397e6748827466d3d with peer 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:03.483119 28882 ts_tablet_manager.cc:927] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a: Initiating tablet copy from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421)
I20250905 19:59:03.485700 28882 tablet_copy_client.cc:323] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Beginning tablet copy session from remote peer at address 127.26.58.129:38421
I20250905 19:59:03.487303 28020 tablet_copy_service.cc:140] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Received BeginTabletCopySession request for tablet fc6a8d5d253643cea8582bef1c4173d6 from peer 7e9c9f591f05464f9b097bf003d14f4a ({username='slave'} at 127.0.0.1:45844)
I20250905 19:59:03.487851 28020 tablet_copy_service.cc:161] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Beginning new tablet copy session on tablet fc6a8d5d253643cea8582bef1c4173d6 from peer 7e9c9f591f05464f9b097bf003d14f4a at {username='slave'} at 127.0.0.1:45844: session id = 7e9c9f591f05464f9b097bf003d14f4a-fc6a8d5d253643cea8582bef1c4173d6
I20250905 19:59:03.494843 28020 tablet_copy_source_session.cc:215] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:03.498312 28882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc6a8d5d253643cea8582bef1c4173d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:03.515632 28882 tablet_copy_client.cc:806] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:03.516209 28882 tablet_copy_client.cc:670] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:03.519797 28882 tablet_copy_client.cc:538] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 19:59:03.527611 28882 tablet_bootstrap.cc:492] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap starting.
I20250905 19:59:03.553928 28882 tablet_bootstrap.cc:492] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:03.554834 28882 tablet_bootstrap.cc:492] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap complete.
I20250905 19:59:03.555394 28882 ts_tablet_manager.cc:1397] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent bootstrapping tablet: real 0.028s user 0.016s sys 0.004s
I20250905 19:59:03.558077 28882 raft_consensus.cc:357] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:03.558807 28882 raft_consensus.cc:738] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Initialized, Role: LEARNER
I20250905 19:59:03.559293 28882 consensus_queue.cc:260] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:03.561512 28882 ts_tablet_manager.cc:1428] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250905 19:59:03.563107 28020 tablet_copy_service.cc:342] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Request end of tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-fc6a8d5d253643cea8582bef1c4173d6 received from {username='slave'} at 127.0.0.1:45844
I20250905 19:59:03.563513 28020 tablet_copy_service.cc:434] P cd764bfeebac4cc9ad851cd4a6a7d3b0: ending tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-fc6a8d5d253643cea8582bef1c4173d6 on tablet fc6a8d5d253643cea8582bef1c4173d6 with peer 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:03.701583 28750 raft_consensus.cc:1215] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250905 19:59:03.808513 28750 raft_consensus.cc:1215] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250905 19:59:03.857895 28750 raft_consensus.cc:1215] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20250905 19:59:04.025162 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 75365.2us
I20250905 19:59:04.100414 28750 raft_consensus.cc:1215] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20250905 19:59:04.148000 28869 raft_consensus.cc:1062] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: attempting to promote NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a to VOTER
I20250905 19:59:04.150763 28869 consensus_queue.cc:237] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:04.172832 28750 raft_consensus.cc:1273] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:04.175361 28154 raft_consensus.cc:1273] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:04.176337 28083 raft_consensus.cc:1273] T 9f7f6e4e3a754dcda90928621168fbe7 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:04.176184 28869 consensus_queue.cc:1035] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:04.180195 28898 consensus_queue.cc:1035] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:04.183063 28898 consensus_queue.cc:1035] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:04.202646 28898 raft_consensus.cc:2953] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:04.204314 28750 raft_consensus.cc:2953] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:04.214349 28080 raft_consensus.cc:2953] T 9f7f6e4e3a754dcda90928621168fbe7 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:04.226254 27907 catalog_manager.cc:5582] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 19:59:04.233213 28155 raft_consensus.cc:2953] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:04.258548 28858 raft_consensus.cc:1062] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9: attempting to promote NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a to VOTER
I20250905 19:59:04.261183 28858 consensus_queue.cc:237] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:04.273378 28155 raft_consensus.cc:1273] T 361ffd5c9a994e0397e6748827466d3d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:04.275007 28001 raft_consensus.cc:1273] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:04.282648 28900 consensus_queue.cc:1035] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:04.287614 28750 raft_consensus.cc:1273] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:04.290019 28893 consensus_queue.cc:1035] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:04.298243 28593 consensus_queue.cc:1035] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:04.351380 28892 raft_consensus.cc:1062] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05: attempting to promote NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a to VOTER
I20250905 19:59:04.354288 28892 consensus_queue.cc:237] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:04.374186 28844 raft_consensus.cc:2953] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:04.383576 28750 raft_consensus.cc:2953] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:04.430263 28752 raft_consensus.cc:1273] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:04.431735 28004 raft_consensus.cc:1273] T 8a8d8393352b4214a20c1683852d4fdd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:04.433228 28892 consensus_queue.cc:1035] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:04.434590 28899 consensus_queue.cc:1035] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:04.463768 28899 raft_consensus.cc:2953] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:04.473062 28750 raft_consensus.cc:2953] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:04.487902 27905 catalog_manager.cc:5582] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:04.493336 28005 raft_consensus.cc:2953] T 8a8d8393352b4214a20c1683852d4fdd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:04.494985 28004 consensus_queue.cc:237] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:04.510490 28154 raft_consensus.cc:1273] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:04.512530 28869 consensus_queue.cc:1035] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:59:04.514508 28750 raft_consensus.cc:1273] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:04.518800 28869 consensus_queue.cc:1035] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 19:59:04.523185 28898 raft_consensus.cc:2953] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:04.534204 28081 raft_consensus.cc:1273] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:04.535619 28750 raft_consensus.cc:2953] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:04.538319 28154 raft_consensus.cc:2953] T 361ffd5c9a994e0397e6748827466d3d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:04.546156 28894 raft_consensus.cc:1062] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0: attempting to promote NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a to VOTER
I20250905 19:59:04.555737 28890 consensus_queue.cc:1035] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:04.596056 28894 consensus_queue.cc:237] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:04.607084 28002 raft_consensus.cc:2953] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:04.623099 28082 raft_consensus.cc:2953] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:04.851172 27908 catalog_manager.cc:5582] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 19:59:04.888517 28154 raft_consensus.cc:1273] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250905 19:59:04.888912 28083 raft_consensus.cc:1273] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250905 19:59:04.890141 28894 consensus_queue.cc:1035] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:59:04.891054 28924 consensus_queue.cc:1035] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:59:04.560835 28153 raft_consensus.cc:2953] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:04.925050 28751 raft_consensus.cc:1273] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 LEARNER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250905 19:59:04.926505 28896 consensus_queue.cc:1035] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 19:59:04.941150 28926 raft_consensus.cc:2953] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:04.942809 28751 raft_consensus.cc:2953] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:04.958513 28151 raft_consensus.cc:2953] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
W20250905 19:59:04.987463 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 373238us
I20250905 19:59:05.011974 27895 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9f7f6e4e3a754dcda90928621168fbe7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 19:59:05.012460 27908 catalog_manager.cc:5582] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:05.018546 27908 catalog_manager.cc:5582] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: config changed from index 3 to 4, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:05.059368 28072 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f7f6e4e3a754dcda90928621168fbe7 with delete_type TABLET_DATA_TOMBSTONED (TS e4413ae5e0b64b52a6f360ac580239b9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59536
I20250905 19:59:05.068902 28081 consensus_queue.cc:237] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:05.075832 28934 tablet_replica.cc:331] T 9f7f6e4e3a754dcda90928621168fbe7 P e4413ae5e0b64b52a6f360ac580239b9: stopping tablet replica
I20250905 19:59:05.076603 28751 raft_consensus.cc:1273] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:05.078486 28006 consensus_queue.cc:237] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:05.066203 28083 raft_consensus.cc:2953] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:05.080160 28921 consensus_queue.cc:1035] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 19:59:05.087301 28934 raft_consensus.cc:2241] T 9f7f6e4e3a754dcda90928621168fbe7 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:05.088222 28934 raft_consensus.cc:2270] T 9f7f6e4e3a754dcda90928621168fbe7 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:05.092836 28751 raft_consensus.cc:1273] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250905 19:59:05.094501 28894 consensus_queue.cc:1035] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250905 19:59:05.095553 28934 ts_tablet_manager.cc:1905] T 9f7f6e4e3a754dcda90928621168fbe7 P e4413ae5e0b64b52a6f360ac580239b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 19:59:05.098556 28916 raft_consensus.cc:2953] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:05.101246 28750 raft_consensus.cc:2953] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:05.107355 28869 raft_consensus.cc:2953] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:05.109387 28751 raft_consensus.cc:2953] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:05.116793 27892 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 361ffd5c9a994e0397e6748827466d3d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 19:59:05.122829 27908 catalog_manager.cc:5582] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: config changed from index 4 to 5, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New cstate: current_term: 3 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:05.127307 28081 raft_consensus.cc:1273] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250905 19:59:05.128793 27895 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet fc6a8d5d253643cea8582bef1c4173d6 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 19:59:05.141916 28145 tablet_service.cc:1515] Processing DeleteTablet for tablet fc6a8d5d253643cea8582bef1c4173d6 with delete_type TABLET_DATA_TOMBSTONED (TS 26b2f874b8e54f08b01a9ed00d5cce05 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33742
I20250905 19:59:05.148331 28934 ts_tablet_manager.cc:1918] T 9f7f6e4e3a754dcda90928621168fbe7 P e4413ae5e0b64b52a6f360ac580239b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250905 19:59:05.151891 28869 consensus_queue.cc:1035] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250905 19:59:05.155972 28934 log.cc:1199] T 9f7f6e4e3a754dcda90928621168fbe7 P e4413ae5e0b64b52a6f360ac580239b9: Deleting WAL directory at /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-1-root/wals/9f7f6e4e3a754dcda90928621168fbe7
I20250905 19:59:05.162588 27892 catalog_manager.cc:4928] TS e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): tablet 9f7f6e4e3a754dcda90928621168fbe7 (table test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]) successfully deleted
I20250905 19:59:05.162860 27906 catalog_manager.cc:5582] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:05.164100 28935 tablet_replica.cc:331] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05: stopping tablet replica
I20250905 19:59:05.166487 28935 raft_consensus.cc:2241] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:05.167405 28935 raft_consensus.cc:2270] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:05.167137 28081 raft_consensus.cc:2953] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:05.171114 28935 ts_tablet_manager.cc:1905] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 19:59:05.178462 28145 tablet_service.cc:1515] Processing DeleteTablet for tablet 361ffd5c9a994e0397e6748827466d3d with delete_type TABLET_DATA_TOMBSTONED (TS 26b2f874b8e54f08b01a9ed00d5cce05 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33742
I20250905 19:59:05.185781 28004 raft_consensus.cc:1273] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:05.191538 28935 ts_tablet_manager.cc:1918] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20250905 19:59:05.192241 28935 log.cc:1199] T fc6a8d5d253643cea8582bef1c4173d6 P 26b2f874b8e54f08b01a9ed00d5cce05: Deleting WAL directory at /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-2-root/wals/fc6a8d5d253643cea8582bef1c4173d6
I20250905 19:59:05.194132 27894 catalog_manager.cc:4928] TS 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): tablet fc6a8d5d253643cea8582bef1c4173d6 (table test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]) successfully deleted
I20250905 19:59:05.195479 28935 tablet_replica.cc:331] T 361ffd5c9a994e0397e6748827466d3d P 26b2f874b8e54f08b01a9ed00d5cce05: stopping tablet replica
I20250905 19:59:05.196720 28935 raft_consensus.cc:2241] T 361ffd5c9a994e0397e6748827466d3d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:05.197623 28935 raft_consensus.cc:2270] T 361ffd5c9a994e0397e6748827466d3d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:05.200827 28919 consensus_queue.cc:1035] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 19:59:05.206368 28006 raft_consensus.cc:2953] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:05.210496 28935 ts_tablet_manager.cc:1905] T 361ffd5c9a994e0397e6748827466d3d P 26b2f874b8e54f08b01a9ed00d5cce05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 19:59:05.247221 28935 ts_tablet_manager.cc:1918] T 361ffd5c9a994e0397e6748827466d3d P 26b2f874b8e54f08b01a9ed00d5cce05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250905 19:59:05.247777 28935 log.cc:1199] T 361ffd5c9a994e0397e6748827466d3d P 26b2f874b8e54f08b01a9ed00d5cce05: Deleting WAL directory at /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-2-root/wals/361ffd5c9a994e0397e6748827466d3d
I20250905 19:59:05.249352 27894 catalog_manager.cc:4928] TS 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): tablet 361ffd5c9a994e0397e6748827466d3d (table test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]) successfully deleted
I20250905 19:59:05.552795 28151 consensus_queue.cc:237] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:05.558516 28750 raft_consensus.cc:1273] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:05.560266 28936 consensus_queue.cc:1035] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:59:05.563148 28079 raft_consensus.cc:1273] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:05.564987 28890 consensus_queue.cc:1035] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:59:05.575284 28750 raft_consensus.cc:2953] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:05.573458 28890 raft_consensus.cc:2953] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:05.586220 27894 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 8a8d8393352b4214a20c1683852d4fdd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 19:59:05.588243 28079 raft_consensus.cc:2953] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:05.588807 27908 catalog_manager.cc:5582] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:05.601590 28000 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a8d8393352b4214a20c1683852d4fdd with delete_type TABLET_DATA_TOMBSTONED (TS cd764bfeebac4cc9ad851cd4a6a7d3b0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54144
W20250905 19:59:05.609474 28106 debug-util.cc:398] Leaking SignalData structure 0x7b080016b040 after lost signal to thread 26861
W20250905 19:59:05.610720 28106 debug-util.cc:398] Leaking SignalData structure 0x7b080016b720 after lost signal to thread 27942
W20250905 19:59:05.612383 28106 debug-util.cc:398] Leaking SignalData structure 0x7b08002bf2a0 after lost signal to thread 28034
W20250905 19:59:05.613893 28106 debug-util.cc:398] Leaking SignalData structure 0x7b08002ef8c0 after lost signal to thread 28109
W20250905 19:59:05.615396 28106 debug-util.cc:398] Leaking SignalData structure 0x7b080030b000 after lost signal to thread 28183
W20250905 19:59:05.616431 28106 debug-util.cc:398] Leaking SignalData structure 0x7b0800041c60 after lost signal to thread 28776
I20250905 19:59:05.619354 28939 tablet_replica.cc:331] T 8a8d8393352b4214a20c1683852d4fdd P cd764bfeebac4cc9ad851cd4a6a7d3b0: stopping tablet replica
I20250905 19:59:05.620772 28939 raft_consensus.cc:2241] T 8a8d8393352b4214a20c1683852d4fdd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:05.621548 28939 raft_consensus.cc:2270] T 8a8d8393352b4214a20c1683852d4fdd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:05.631498 28939 ts_tablet_manager.cc:1905] T 8a8d8393352b4214a20c1683852d4fdd P cd764bfeebac4cc9ad851cd4a6a7d3b0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 19:59:05.646610 28939 ts_tablet_manager.cc:1918] T 8a8d8393352b4214a20c1683852d4fdd P cd764bfeebac4cc9ad851cd4a6a7d3b0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250905 19:59:05.646960 28939 log.cc:1199] T 8a8d8393352b4214a20c1683852d4fdd P cd764bfeebac4cc9ad851cd4a6a7d3b0: Deleting WAL directory at /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-0-root/wals/8a8d8393352b4214a20c1683852d4fdd
I20250905 19:59:05.648483 27895 catalog_manager.cc:4928] TS cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): tablet 8a8d8393352b4214a20c1683852d4fdd (table test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]) successfully deleted
I20250905 19:59:06.711225 28151 consensus_queue.cc:237] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
W20250905 19:59:06.720849 28123 consensus_peers.cc:489] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 -> Peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299): Couldn't send request to peer 7e9c9f591f05464f9b097bf003d14f4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd2b679c53994263846f51ceec97a150. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:06.720937 28079 raft_consensus.cc:1273] T fd2b679c53994263846f51ceec97a150 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:06.722832 28004 raft_consensus.cc:1273] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:06.724637 28936 consensus_queue.cc:1035] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:06.726683 28936 consensus_queue.cc:1035] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:06.754326 28855 raft_consensus.cc:2953] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:06.759672 28010 raft_consensus.cc:2953] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:06.775341 28079 raft_consensus.cc:2953] T fd2b679c53994263846f51ceec97a150 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:06.787703 28004 consensus_queue.cc:237] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:06.791286 27907 catalog_manager.cc:5582] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250905 19:59:06.809389 27974 consensus_peers.cc:489] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 -> Peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299): Couldn't send request to peer 7e9c9f591f05464f9b097bf003d14f4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8e0ac2dfe59d423189dd8d72eee98a42. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:06.813047 28159 raft_consensus.cc:1273] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:06.814678 28869 consensus_queue.cc:1035] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:06.824254 28079 raft_consensus.cc:1273] T 8e0ac2dfe59d423189dd8d72eee98a42 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:06.825186 28947 raft_consensus.cc:2953] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:06.827652 28151 raft_consensus.cc:2953] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:06.848479 28869 consensus_queue.cc:1035] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:06.858721 28079 consensus_queue.cc:237] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:06.862946 27906 catalog_manager.cc:5582] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
W20250905 19:59:06.895713 28049 consensus_peers.cc:489] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 -> Peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299): Couldn't send request to peer 7e9c9f591f05464f9b097bf003d14f4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aace14abd237420c8211ce611ea09f0e. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:06.898274 28082 raft_consensus.cc:2953] T 8e0ac2dfe59d423189dd8d72eee98a42 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:06.916517 28004 raft_consensus.cc:1273] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:06.919095 28945 consensus_queue.cc:1035] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20250905 19:59:06.921159 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 44514.8us
I20250905 19:59:06.925725 28159 raft_consensus.cc:1273] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:06.932736 28900 consensus_queue.cc:1035] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:06.941864 28944 raft_consensus.cc:2953] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:06.945690 28004 raft_consensus.cc:2953] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:06.960369 28151 raft_consensus.cc:2953] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:06.966002 28082 consensus_queue.cc:237] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:06.969920 27908 catalog_manager.cc:5582] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
W20250905 19:59:06.983431 28049 consensus_peers.cc:489] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 -> Peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299): Couldn't send request to peer 7e9c9f591f05464f9b097bf003d14f4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fba42101cd094077aa85e3f9186366e7. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:06.985698 28005 raft_consensus.cc:1273] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:06.989424 28159 raft_consensus.cc:1273] T fba42101cd094077aa85e3f9186366e7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:06.995455 28916 consensus_queue.cc:1035] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:06.996626 28893 consensus_queue.cc:1035] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:07.011547 28893 raft_consensus.cc:2953] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:07.025305 28159 raft_consensus.cc:2953] T fba42101cd094077aa85e3f9186366e7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:07.027081 28002 raft_consensus.cc:2953] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:07.062615 27906 catalog_manager.cc:5582] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:07.297802 28965 ts_tablet_manager.cc:927] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a: Initiating tablet copy from peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:07.303514 28965 tablet_copy_client.cc:323] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Beginning tablet copy session from remote peer at address 127.26.58.131:36705
I20250905 19:59:07.306150 28169 tablet_copy_service.cc:140] P 26b2f874b8e54f08b01a9ed00d5cce05: Received BeginTabletCopySession request for tablet fd2b679c53994263846f51ceec97a150 from peer 7e9c9f591f05464f9b097bf003d14f4a ({username='slave'} at 127.0.0.1:39586)
I20250905 19:59:07.306927 28169 tablet_copy_service.cc:161] P 26b2f874b8e54f08b01a9ed00d5cce05: Beginning new tablet copy session on tablet fd2b679c53994263846f51ceec97a150 from peer 7e9c9f591f05464f9b097bf003d14f4a at {username='slave'} at 127.0.0.1:39586: session id = 7e9c9f591f05464f9b097bf003d14f4a-fd2b679c53994263846f51ceec97a150
I20250905 19:59:07.316663 28169 tablet_copy_source_session.cc:215] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:07.320693 28965 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd2b679c53994263846f51ceec97a150. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:07.346400 28965 tablet_copy_client.cc:806] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:07.347276 28965 tablet_copy_client.cc:670] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:07.352442 28965 tablet_copy_client.cc:538] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 19:59:07.365167 28965 tablet_bootstrap.cc:492] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap starting.
I20250905 19:59:07.373515 28968 ts_tablet_manager.cc:927] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a: Initiating tablet copy from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421)
I20250905 19:59:07.378768 28968 tablet_copy_client.cc:323] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Beginning tablet copy session from remote peer at address 127.26.58.129:38421
I20250905 19:59:07.380921 28020 tablet_copy_service.cc:140] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Received BeginTabletCopySession request for tablet 8e0ac2dfe59d423189dd8d72eee98a42 from peer 7e9c9f591f05464f9b097bf003d14f4a ({username='slave'} at 127.0.0.1:45844)
I20250905 19:59:07.381570 28020 tablet_copy_service.cc:161] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Beginning new tablet copy session on tablet 8e0ac2dfe59d423189dd8d72eee98a42 from peer 7e9c9f591f05464f9b097bf003d14f4a at {username='slave'} at 127.0.0.1:45844: session id = 7e9c9f591f05464f9b097bf003d14f4a-8e0ac2dfe59d423189dd8d72eee98a42
I20250905 19:59:07.389405 28967 ts_tablet_manager.cc:927] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a: Initiating tablet copy from peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:07.392242 28020 tablet_copy_source_session.cc:215] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:07.403417 28968 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e0ac2dfe59d423189dd8d72eee98a42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:07.428598 28965 tablet_bootstrap.cc:492] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:07.429651 28965 tablet_bootstrap.cc:492] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap complete.
I20250905 19:59:07.430416 28965 ts_tablet_manager.cc:1397] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent bootstrapping tablet: real 0.066s user 0.029s sys 0.004s
I20250905 19:59:07.441432 28965 raft_consensus.cc:357] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:07.442360 28965 raft_consensus.cc:738] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Initialized, Role: LEARNER
I20250905 19:59:07.443994 28968 tablet_copy_client.cc:806] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:07.444689 28968 tablet_copy_client.cc:670] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:07.443099 28965 consensus_queue.cc:260] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:07.456930 28968 tablet_copy_client.cc:538] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 19:59:07.468705 28968 tablet_bootstrap.cc:492] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap starting.
I20250905 19:59:07.481549 28965 ts_tablet_manager.cc:1428] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent starting tablet: real 0.051s user 0.009s sys 0.004s
I20250905 19:59:07.486421 28169 tablet_copy_service.cc:342] P 26b2f874b8e54f08b01a9ed00d5cce05: Request end of tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-fd2b679c53994263846f51ceec97a150 received from {username='slave'} at 127.0.0.1:39586
I20250905 19:59:07.487054 28169 tablet_copy_service.cc:434] P 26b2f874b8e54f08b01a9ed00d5cce05: ending tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-fd2b679c53994263846f51ceec97a150 on tablet fd2b679c53994263846f51ceec97a150 with peer 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:07.527724 28968 tablet_bootstrap.cc:492] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:07.529711 28968 tablet_bootstrap.cc:492] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap complete.
I20250905 19:59:07.530526 28968 ts_tablet_manager.cc:1397] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent bootstrapping tablet: real 0.062s user 0.029s sys 0.004s
I20250905 19:59:07.533663 28968 raft_consensus.cc:357] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:07.534595 28968 raft_consensus.cc:738] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Initialized, Role: LEARNER
I20250905 19:59:07.535310 28968 consensus_queue.cc:260] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:07.537976 28968 ts_tablet_manager.cc:1428] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250905 19:59:07.540202 28020 tablet_copy_service.cc:342] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Request end of tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-8e0ac2dfe59d423189dd8d72eee98a42 received from {username='slave'} at 127.0.0.1:45844
I20250905 19:59:07.540726 28020 tablet_copy_service.cc:434] P cd764bfeebac4cc9ad851cd4a6a7d3b0: ending tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-8e0ac2dfe59d423189dd8d72eee98a42 on tablet 8e0ac2dfe59d423189dd8d72eee98a42 with peer 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:07.541258 28967 tablet_copy_client.cc:323] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Beginning tablet copy session from remote peer at address 127.26.58.130:34193
I20250905 19:59:07.543287 28095 tablet_copy_service.cc:140] P e4413ae5e0b64b52a6f360ac580239b9: Received BeginTabletCopySession request for tablet aace14abd237420c8211ce611ea09f0e from peer 7e9c9f591f05464f9b097bf003d14f4a ({username='slave'} at 127.0.0.1:50660)
I20250905 19:59:07.543872 28095 tablet_copy_service.cc:161] P e4413ae5e0b64b52a6f360ac580239b9: Beginning new tablet copy session on tablet aace14abd237420c8211ce611ea09f0e from peer 7e9c9f591f05464f9b097bf003d14f4a at {username='slave'} at 127.0.0.1:50660: session id = 7e9c9f591f05464f9b097bf003d14f4a-aace14abd237420c8211ce611ea09f0e
I20250905 19:59:07.550958 28968 ts_tablet_manager.cc:927] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a: Initiating tablet copy from peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:07.552838 28095 tablet_copy_source_session.cc:215] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:07.557951 28967 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aace14abd237420c8211ce611ea09f0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:07.567333 28968 tablet_copy_client.cc:323] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Beginning tablet copy session from remote peer at address 127.26.58.130:34193
I20250905 19:59:07.575951 28095 tablet_copy_service.cc:140] P e4413ae5e0b64b52a6f360ac580239b9: Received BeginTabletCopySession request for tablet fba42101cd094077aa85e3f9186366e7 from peer 7e9c9f591f05464f9b097bf003d14f4a ({username='slave'} at 127.0.0.1:50660)
I20250905 19:59:07.576589 28095 tablet_copy_service.cc:161] P e4413ae5e0b64b52a6f360ac580239b9: Beginning new tablet copy session on tablet fba42101cd094077aa85e3f9186366e7 from peer 7e9c9f591f05464f9b097bf003d14f4a at {username='slave'} at 127.0.0.1:50660: session id = 7e9c9f591f05464f9b097bf003d14f4a-fba42101cd094077aa85e3f9186366e7
I20250905 19:59:07.593273 28967 tablet_copy_client.cc:806] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:07.593961 28967 tablet_copy_client.cc:670] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:07.603339 28095 tablet_copy_source_session.cc:215] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:07.605105 28967 tablet_copy_client.cc:538] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 19:59:07.607662 28968 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fba42101cd094077aa85e3f9186366e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:07.615483 28967 tablet_bootstrap.cc:492] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap starting.
I20250905 19:59:07.635075 28968 tablet_copy_client.cc:806] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:07.635912 28968 tablet_copy_client.cc:670] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:07.640677 28968 tablet_copy_client.cc:538] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 19:59:07.647878 28967 tablet_bootstrap.cc:492] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:07.648888 28967 tablet_bootstrap.cc:492] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap complete.
I20250905 19:59:07.649771 28967 ts_tablet_manager.cc:1397] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a: Time spent bootstrapping tablet: real 0.035s user 0.018s sys 0.012s
I20250905 19:59:07.652467 28968 tablet_bootstrap.cc:492] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap starting.
I20250905 19:59:07.652928 28967 raft_consensus.cc:357] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:07.653821 28967 raft_consensus.cc:738] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Initialized, Role: LEARNER
I20250905 19:59:07.654750 28967 consensus_queue.cc:260] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:07.657727 28967 ts_tablet_manager.cc:1428] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250905 19:59:07.663095 28095 tablet_copy_service.cc:342] P e4413ae5e0b64b52a6f360ac580239b9: Request end of tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-aace14abd237420c8211ce611ea09f0e received from {username='slave'} at 127.0.0.1:50660
I20250905 19:59:07.663676 28095 tablet_copy_service.cc:434] P e4413ae5e0b64b52a6f360ac580239b9: ending tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-aace14abd237420c8211ce611ea09f0e on tablet aace14abd237420c8211ce611ea09f0e with peer 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:07.685771 28968 tablet_bootstrap.cc:492] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:07.686914 28968 tablet_bootstrap.cc:492] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap complete.
I20250905 19:59:07.688091 28968 ts_tablet_manager.cc:1397] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent bootstrapping tablet: real 0.036s user 0.024s sys 0.007s
I20250905 19:59:07.692739 28968 raft_consensus.cc:357] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:07.693838 28968 raft_consensus.cc:738] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Initialized, Role: LEARNER
I20250905 19:59:07.694592 28968 consensus_queue.cc:260] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:07.701004 28968 ts_tablet_manager.cc:1428] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20250905 19:59:07.703225 28095 tablet_copy_service.cc:342] P e4413ae5e0b64b52a6f360ac580239b9: Request end of tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-fba42101cd094077aa85e3f9186366e7 received from {username='slave'} at 127.0.0.1:50660
I20250905 19:59:07.703751 28095 tablet_copy_service.cc:434] P e4413ae5e0b64b52a6f360ac580239b9: ending tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-fba42101cd094077aa85e3f9186366e7 on tablet fba42101cd094077aa85e3f9186366e7 with peer 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:07.755332 28751 raft_consensus.cc:1215] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250905 19:59:07.833968 28751 raft_consensus.cc:1215] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250905 19:59:07.862304 28751 raft_consensus.cc:1215] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250905 19:59:08.120208 28751 raft_consensus.cc:1215] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250905 19:59:08.153437 28869 raft_consensus.cc:1062] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0: attempting to promote NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a to VOTER
I20250905 19:59:08.155910 28869 consensus_queue.cc:237] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:08.163314 28751 raft_consensus.cc:1273] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:08.164837 28947 consensus_queue.cc:1035] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:08.167974 28083 raft_consensus.cc:1273] T 8e0ac2dfe59d423189dd8d72eee98a42 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:08.169792 28150 raft_consensus.cc:1273] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:08.170197 28947 consensus_queue.cc:1035] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:08.176992 28946 consensus_queue.cc:1035] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:08.191051 28947 raft_consensus.cc:2953] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.195928 28079 raft_consensus.cc:2953] T 8e0ac2dfe59d423189dd8d72eee98a42 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.204427 28751 raft_consensus.cc:2953] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.196774 28150 raft_consensus.cc:2953] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.215803 27906 catalog_manager.cc:5582] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:08.282634 28002 consensus_queue.cc:237] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:08.289148 28749 raft_consensus.cc:1273] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:08.289438 28150 raft_consensus.cc:1273] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:08.291010 28947 consensus_queue.cc:1035] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:59:08.293172 28946 consensus_queue.cc:1035] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:59:08.298893 28947 raft_consensus.cc:2953] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.302592 28151 raft_consensus.cc:2953] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.302593 28749 raft_consensus.cc:2953] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.316761 27895 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 8e0ac2dfe59d423189dd8d72eee98a42 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 19:59:08.323964 27908 catalog_manager.cc:5582] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.351857 28945 raft_consensus.cc:1062] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9: attempting to promote NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a to VOTER
I20250905 19:59:08.354527 28945 consensus_queue.cc:237] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:08.357934 28072 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e0ac2dfe59d423189dd8d72eee98a42 with delete_type TABLET_DATA_TOMBSTONED (TS e4413ae5e0b64b52a6f360ac580239b9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59536
I20250905 19:59:08.373101 28985 tablet_replica.cc:331] T 8e0ac2dfe59d423189dd8d72eee98a42 P e4413ae5e0b64b52a6f360ac580239b9: stopping tablet replica
I20250905 19:59:08.374418 28985 raft_consensus.cc:2241] T 8e0ac2dfe59d423189dd8d72eee98a42 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:08.375303 28985 raft_consensus.cc:2270] T 8e0ac2dfe59d423189dd8d72eee98a42 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:08.380626 28985 ts_tablet_manager.cc:1905] T 8e0ac2dfe59d423189dd8d72eee98a42 P e4413ae5e0b64b52a6f360ac580239b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 19:59:08.386220 28749 raft_consensus.cc:1273] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:08.387117 28002 raft_consensus.cc:1273] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:08.388310 28945 consensus_queue.cc:1035] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:08.390307 28893 consensus_queue.cc:1035] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:08.397439 28150 raft_consensus.cc:1273] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:08.401530 28945 consensus_queue.cc:1035] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:08.434757 28985 ts_tablet_manager.cc:1918] T 8e0ac2dfe59d423189dd8d72eee98a42 P e4413ae5e0b64b52a6f360ac580239b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250905 19:59:08.435310 28985 log.cc:1199] T 8e0ac2dfe59d423189dd8d72eee98a42 P e4413ae5e0b64b52a6f360ac580239b9: Deleting WAL directory at /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-1-root/wals/8e0ac2dfe59d423189dd8d72eee98a42
I20250905 19:59:08.439342 28893 raft_consensus.cc:2953] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.443048 27892 catalog_manager.cc:4928] TS e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): tablet 8e0ac2dfe59d423189dd8d72eee98a42 (table test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]) successfully deleted
I20250905 19:59:08.460223 28010 raft_consensus.cc:2953] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.466161 28749 raft_consensus.cc:2953] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.468997 27907 catalog_manager.cc:5582] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:08.492574 28150 raft_consensus.cc:2953] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.499771 28948 raft_consensus.cc:1062] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05: attempting to promote NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a to VOTER
I20250905 19:59:08.510756 28948 consensus_queue.cc:237] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:08.548256 28751 raft_consensus.cc:1273] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:08.550001 28990 consensus_queue.cc:1035] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:08.552392 28079 raft_consensus.cc:1273] T fd2b679c53994263846f51ceec97a150 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:08.554605 28990 consensus_queue.cc:1035] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:08.563714 28010 raft_consensus.cc:1273] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:08.566931 28990 consensus_queue.cc:1035] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:08.555928 28083 consensus_queue.cc:237] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:08.594969 28948 raft_consensus.cc:2953] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.600781 28749 raft_consensus.cc:2953] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.607020 28751 raft_consensus.cc:1273] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:08.610175 28992 consensus_queue.cc:1035] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 19:59:08.613862 28150 raft_consensus.cc:1273] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:08.620218 28991 raft_consensus.cc:2953] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.624516 28893 raft_consensus.cc:1062] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9: attempting to promote NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a to VOTER
I20250905 19:59:08.623800 28750 raft_consensus.cc:2953] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.627130 28893 consensus_queue.cc:237] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:08.639427 28992 consensus_queue.cc:1035] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 19:59:08.642696 28082 raft_consensus.cc:2953] T fd2b679c53994263846f51ceec97a150 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.652323 27907 catalog_manager.cc:5582] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 19:59:08.675132 28000 tablet_service.cc:1515] Processing DeleteTablet for tablet aace14abd237420c8211ce611ea09f0e with delete_type TABLET_DATA_TOMBSTONED (TS cd764bfeebac4cc9ad851cd4a6a7d3b0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54144
I20250905 19:59:08.679661 27907 catalog_manager.cc:5582] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.689318 28010 raft_consensus.cc:2953] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.698771 29000 tablet_replica.cc:331] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0: stopping tablet replica
I20250905 19:59:08.700213 29000 raft_consensus.cc:2241] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:08.700363 28002 raft_consensus.cc:1273] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:08.701200 29000 raft_consensus.cc:2270] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:08.702133 28916 consensus_queue.cc:1035] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:08.716128 29000 ts_tablet_manager.cc:1905] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 19:59:08.722378 28750 raft_consensus.cc:1273] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:08.725765 28992 consensus_queue.cc:1035] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:08.733690 27892 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet aace14abd237420c8211ce611ea09f0e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 19:59:08.766276 28153 raft_consensus.cc:1273] T fba42101cd094077aa85e3f9186366e7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:08.772832 28150 consensus_queue.cc:237] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:08.758699 28151 raft_consensus.cc:2953] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.801846 28005 raft_consensus.cc:1273] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:08.803131 28992 consensus_queue.cc:1035] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:08.808508 29000 ts_tablet_manager.cc:1918] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250905 19:59:08.809089 29000 log.cc:1199] T aace14abd237420c8211ce611ea09f0e P cd764bfeebac4cc9ad851cd4a6a7d3b0: Deleting WAL directory at /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-0-root/wals/aace14abd237420c8211ce611ea09f0e
I20250905 19:59:08.812552 27895 catalog_manager.cc:4928] TS cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): tablet aace14abd237420c8211ce611ea09f0e (table test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]) successfully deleted
I20250905 19:59:08.814196 28749 raft_consensus.cc:1273] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:08.818756 28999 consensus_queue.cc:1035] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 19:59:08.823102 28990 consensus_queue.cc:1035] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 19:59:08.839068 28893 raft_consensus.cc:2953] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.843307 29005 raft_consensus.cc:2953] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.844368 28750 raft_consensus.cc:2953] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.860004 28010 raft_consensus.cc:2953] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.864053 28749 raft_consensus.cc:2953] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.866901 27894 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet fd2b679c53994263846f51ceec97a150 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 19:59:08.879984 28010 raft_consensus.cc:2953] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.889695 27907 catalog_manager.cc:5582] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.893065 27908 catalog_manager.cc:5582] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.916283 28072 tablet_service.cc:1515] Processing DeleteTablet for tablet fd2b679c53994263846f51ceec97a150 with delete_type TABLET_DATA_TOMBSTONED (TS e4413ae5e0b64b52a6f360ac580239b9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59536
I20250905 19:59:08.914884 28159 raft_consensus.cc:2953] T fba42101cd094077aa85e3f9186366e7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.917181 28985 tablet_replica.cc:331] T fd2b679c53994263846f51ceec97a150 P e4413ae5e0b64b52a6f360ac580239b9: stopping tablet replica
I20250905 19:59:08.918310 28985 raft_consensus.cc:2241] T fd2b679c53994263846f51ceec97a150 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:08.919070 28985 raft_consensus.cc:2270] T fd2b679c53994263846f51ceec97a150 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:08.922719 28985 ts_tablet_manager.cc:1905] T fd2b679c53994263846f51ceec97a150 P e4413ae5e0b64b52a6f360ac580239b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 19:59:08.928280 28082 consensus_queue.cc:237] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:08.951676 28005 raft_consensus.cc:1273] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:08.954070 28916 consensus_queue.cc:1035] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 19:59:08.964319 28916 raft_consensus.cc:2953] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.966038 28005 raft_consensus.cc:2953] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:08.985729 28985 ts_tablet_manager.cc:1918] T fd2b679c53994263846f51ceec97a150 P e4413ae5e0b64b52a6f360ac580239b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250905 19:59:08.986382 28985 log.cc:1199] T fd2b679c53994263846f51ceec97a150 P e4413ae5e0b64b52a6f360ac580239b9: Deleting WAL directory at /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-1-root/wals/fd2b679c53994263846f51ceec97a150
I20250905 19:59:08.991511 27892 catalog_manager.cc:4928] TS e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): tablet fd2b679c53994263846f51ceec97a150 (table test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]) successfully deleted
I20250905 19:59:08.982944 28749 raft_consensus.cc:1273] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:08.999084 28945 consensus_queue.cc:1035] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20250905 19:59:09.011111 28750 raft_consensus.cc:2953] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:09.016919 27892 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet fba42101cd094077aa85e3f9186366e7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 19:59:09.030225 27905 catalog_manager.cc:5582] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:09.065744 28145 tablet_service.cc:1515] Processing DeleteTablet for tablet fba42101cd094077aa85e3f9186366e7 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 4)) from {username='slave'} at 127.0.0.1:33742
I20250905 19:59:09.077308 29011 tablet_replica.cc:331] T fba42101cd094077aa85e3f9186366e7 P 26b2f874b8e54f08b01a9ed00d5cce05: stopping tablet replica
I20250905 19:59:09.078516 29011 raft_consensus.cc:2241] T fba42101cd094077aa85e3f9186366e7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:09.079108 29011 raft_consensus.cc:2270] T fba42101cd094077aa85e3f9186366e7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:09.083251 29011 ts_tablet_manager.cc:1905] T fba42101cd094077aa85e3f9186366e7 P 26b2f874b8e54f08b01a9ed00d5cce05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 19:59:09.100000 29011 ts_tablet_manager.cc:1918] T fba42101cd094077aa85e3f9186366e7 P 26b2f874b8e54f08b01a9ed00d5cce05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250905 19:59:09.100370 29011 log.cc:1199] T fba42101cd094077aa85e3f9186366e7 P 26b2f874b8e54f08b01a9ed00d5cce05: Deleting WAL directory at /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-2-root/wals/fba42101cd094077aa85e3f9186366e7
I20250905 19:59:09.101984 27894 catalog_manager.cc:4928] TS 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): tablet fba42101cd094077aa85e3f9186366e7 (table test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]) successfully deleted
I20250905 19:59:10.147258 28010 consensus_queue.cc:237] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:10.153906 28159 raft_consensus.cc:1273] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250905 19:59:10.154251 27974 consensus_peers.cc:489] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 -> Peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299): Couldn't send request to peer 7e9c9f591f05464f9b097bf003d14f4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 18ed409dabfc45b1843e7d8a316a047d. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:10.155151 28079 raft_consensus.cc:1273] T 18ed409dabfc45b1843e7d8a316a047d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:10.155354 28869 consensus_queue.cc:1035] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:10.156450 28947 consensus_queue.cc:1035] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:10.165728 28869 raft_consensus.cc:2953] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:10.167945 28159 raft_consensus.cc:2953] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:10.191843 28079 raft_consensus.cc:2953] T 18ed409dabfc45b1843e7d8a316a047d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:10.194799 28010 consensus_queue.cc:237] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:10.231348 28079 raft_consensus.cc:1273] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:10.232877 29016 consensus_queue.cc:1035] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250905 19:59:10.237741 27974 consensus_peers.cc:489] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 -> Peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299): Couldn't send request to peer 7e9c9f591f05464f9b097bf003d14f4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62ccdb07048e4762b1b31c36b14610f5. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:10.221920 27908 catalog_manager.cc:5582] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
W20250905 19:59:10.255328 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 68092.6us
I20250905 19:59:10.258520 28151 raft_consensus.cc:1273] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:10.261673 28947 raft_consensus.cc:2953] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:10.271023 29016 consensus_queue.cc:1035] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:10.274417 28079 raft_consensus.cc:2953] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:10.285583 28159 consensus_queue.cc:237] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:10.306330 28151 raft_consensus.cc:2953] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:10.312155 28079 raft_consensus.cc:1273] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:10.314848 28990 consensus_queue.cc:1035] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250905 19:59:10.318367 28123 consensus_peers.cc:489] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 -> Peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299): Couldn't send request to peer 7e9c9f591f05464f9b097bf003d14f4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0adebfdecbae47298f234b849fcea7fc. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:10.321161 28004 raft_consensus.cc:1273] T 0adebfdecbae47298f234b849fcea7fc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:10.343883 28990 raft_consensus.cc:2953] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:10.347131 28082 raft_consensus.cc:2953] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:10.352789 27907 catalog_manager.cc:5582] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250905 19:59:10.408936 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 86734.4us
I20250905 19:59:10.428022 27907 catalog_manager.cc:5582] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:10.430953 29029 consensus_queue.cc:1035] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:10.481504 28005 raft_consensus.cc:2953] T 0adebfdecbae47298f234b849fcea7fc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:10.499894 28150 consensus_queue.cc:237] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
W20250905 19:59:10.507300 28123 consensus_peers.cc:489] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 -> Peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299): Couldn't send request to peer 7e9c9f591f05464f9b097bf003d14f4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0889204295e4daa99d81e5b2a45ac1c. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:10.510103 28082 raft_consensus.cc:1273] T c0889204295e4daa99d81e5b2a45ac1c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:10.511353 28005 raft_consensus.cc:1273] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:10.511828 29031 consensus_queue.cc:1035] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:10.515192 29029 consensus_queue.cc:1035] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20250905 19:59:10.542758 29032 raft_consensus.cc:2953] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:10.544276 28004 raft_consensus.cc:2953] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:10.544261 28082 raft_consensus.cc:2953] T c0889204295e4daa99d81e5b2a45ac1c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:10.560657 27905 catalog_manager.cc:5582] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:10.701536 29037 ts_tablet_manager.cc:927] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a: Initiating tablet copy from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421)
I20250905 19:59:10.707383 29038 ts_tablet_manager.cc:927] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a: Initiating tablet copy from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421)
I20250905 19:59:10.722796 29037 tablet_copy_client.cc:323] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Beginning tablet copy session from remote peer at address 127.26.58.129:38421
I20250905 19:59:10.724701 28020 tablet_copy_service.cc:140] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Received BeginTabletCopySession request for tablet 18ed409dabfc45b1843e7d8a316a047d from peer 7e9c9f591f05464f9b097bf003d14f4a ({username='slave'} at 127.0.0.1:45844)
I20250905 19:59:10.725307 28020 tablet_copy_service.cc:161] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Beginning new tablet copy session on tablet 18ed409dabfc45b1843e7d8a316a047d from peer 7e9c9f591f05464f9b097bf003d14f4a at {username='slave'} at 127.0.0.1:45844: session id = 7e9c9f591f05464f9b097bf003d14f4a-18ed409dabfc45b1843e7d8a316a047d
I20250905 19:59:10.727691 29038 tablet_copy_client.cc:323] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Beginning tablet copy session from remote peer at address 127.26.58.129:38421
I20250905 19:59:10.729416 28019 tablet_copy_service.cc:140] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Received BeginTabletCopySession request for tablet 62ccdb07048e4762b1b31c36b14610f5 from peer 7e9c9f591f05464f9b097bf003d14f4a ({username='slave'} at 127.0.0.1:45844)
I20250905 19:59:10.729977 28019 tablet_copy_service.cc:161] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Beginning new tablet copy session on tablet 62ccdb07048e4762b1b31c36b14610f5 from peer 7e9c9f591f05464f9b097bf003d14f4a at {username='slave'} at 127.0.0.1:45844: session id = 7e9c9f591f05464f9b097bf003d14f4a-62ccdb07048e4762b1b31c36b14610f5
I20250905 19:59:10.734019 28020 tablet_copy_source_session.cc:215] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:10.737809 29037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18ed409dabfc45b1843e7d8a316a047d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:10.737893 28019 tablet_copy_source_session.cc:215] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:10.741268 29038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62ccdb07048e4762b1b31c36b14610f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:10.762503 29038 tablet_copy_client.cc:806] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:10.763200 29038 tablet_copy_client.cc:670] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:10.767127 29038 tablet_copy_client.cc:538] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 19:59:10.777323 29037 tablet_copy_client.cc:806] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:10.778532 29038 tablet_bootstrap.cc:492] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap starting.
I20250905 19:59:10.780006 29037 tablet_copy_client.cc:670] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:10.787111 29037 tablet_copy_client.cc:538] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 19:59:10.811224 29037 tablet_bootstrap.cc:492] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap starting.
I20250905 19:59:10.825676 29038 tablet_bootstrap.cc:492] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:10.827889 29038 tablet_bootstrap.cc:492] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap complete.
I20250905 19:59:10.828877 29038 ts_tablet_manager.cc:1397] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent bootstrapping tablet: real 0.051s user 0.011s sys 0.014s
I20250905 19:59:10.832016 29038 raft_consensus.cc:357] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:10.833002 29038 raft_consensus.cc:738] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Initialized, Role: LEARNER
I20250905 19:59:10.833730 29038 consensus_queue.cc:260] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:10.843436 29037 tablet_bootstrap.cc:492] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:10.844637 29037 tablet_bootstrap.cc:492] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap complete.
I20250905 19:59:10.845485 29037 ts_tablet_manager.cc:1397] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a: Time spent bootstrapping tablet: real 0.035s user 0.021s sys 0.010s
I20250905 19:59:10.848990 29037 raft_consensus.cc:357] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:10.850127 29037 raft_consensus.cc:738] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Initialized, Role: LEARNER
I20250905 19:59:10.851836 29037 consensus_queue.cc:260] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:10.861869 29037 ts_tablet_manager.cc:1428] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20250905 19:59:10.854074 29038 ts_tablet_manager.cc:1428] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent starting tablet: real 0.025s user 0.013s sys 0.004s
I20250905 19:59:10.863827 28019 tablet_copy_service.cc:342] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Request end of tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-18ed409dabfc45b1843e7d8a316a047d received from {username='slave'} at 127.0.0.1:45844
I20250905 19:59:10.864269 28019 tablet_copy_service.cc:434] P cd764bfeebac4cc9ad851cd4a6a7d3b0: ending tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-18ed409dabfc45b1843e7d8a316a047d on tablet 18ed409dabfc45b1843e7d8a316a047d with peer 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:10.866837 28019 tablet_copy_service.cc:342] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Request end of tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-62ccdb07048e4762b1b31c36b14610f5 received from {username='slave'} at 127.0.0.1:45844
I20250905 19:59:10.867295 28019 tablet_copy_service.cc:434] P cd764bfeebac4cc9ad851cd4a6a7d3b0: ending tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-62ccdb07048e4762b1b31c36b14610f5 on tablet 62ccdb07048e4762b1b31c36b14610f5 with peer 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:10.885598 29037 ts_tablet_manager.cc:927] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a: Initiating tablet copy from peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:10.887281 29037 tablet_copy_client.cc:323] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Beginning tablet copy session from remote peer at address 127.26.58.131:36705
I20250905 19:59:10.888811 28169 tablet_copy_service.cc:140] P 26b2f874b8e54f08b01a9ed00d5cce05: Received BeginTabletCopySession request for tablet 0adebfdecbae47298f234b849fcea7fc from peer 7e9c9f591f05464f9b097bf003d14f4a ({username='slave'} at 127.0.0.1:39586)
I20250905 19:59:10.889405 28169 tablet_copy_service.cc:161] P 26b2f874b8e54f08b01a9ed00d5cce05: Beginning new tablet copy session on tablet 0adebfdecbae47298f234b849fcea7fc from peer 7e9c9f591f05464f9b097bf003d14f4a at {username='slave'} at 127.0.0.1:39586: session id = 7e9c9f591f05464f9b097bf003d14f4a-0adebfdecbae47298f234b849fcea7fc
I20250905 19:59:10.896494 28169 tablet_copy_source_session.cc:215] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:10.900384 29037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0adebfdecbae47298f234b849fcea7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:10.908510 29038 ts_tablet_manager.cc:927] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a: Initiating tablet copy from peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:10.956624 29038 tablet_copy_client.cc:323] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Beginning tablet copy session from remote peer at address 127.26.58.131:36705
I20250905 19:59:10.984902 29037 tablet_copy_client.cc:806] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:10.985770 29037 tablet_copy_client.cc:670] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:11.010089 28169 tablet_copy_service.cc:140] P 26b2f874b8e54f08b01a9ed00d5cce05: Received BeginTabletCopySession request for tablet c0889204295e4daa99d81e5b2a45ac1c from peer 7e9c9f591f05464f9b097bf003d14f4a ({username='slave'} at 127.0.0.1:39586)
I20250905 19:59:11.010749 28169 tablet_copy_service.cc:161] P 26b2f874b8e54f08b01a9ed00d5cce05: Beginning new tablet copy session on tablet c0889204295e4daa99d81e5b2a45ac1c from peer 7e9c9f591f05464f9b097bf003d14f4a at {username='slave'} at 127.0.0.1:39586: session id = 7e9c9f591f05464f9b097bf003d14f4a-c0889204295e4daa99d81e5b2a45ac1c
I20250905 19:59:11.018999 28169 tablet_copy_source_session.cc:215] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:11.037180 29037 tablet_copy_client.cc:538] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 19:59:11.040331 29038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0889204295e4daa99d81e5b2a45ac1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:11.062000 29038 tablet_copy_client.cc:806] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:11.062800 29038 tablet_copy_client.cc:670] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:11.069239 29038 tablet_copy_client.cc:538] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 19:59:11.072791 29037 tablet_bootstrap.cc:492] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap starting.
I20250905 19:59:11.093303 29038 tablet_bootstrap.cc:492] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap starting.
I20250905 19:59:11.110311 29037 tablet_bootstrap.cc:492] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:11.111380 29037 tablet_bootstrap.cc:492] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap complete.
I20250905 19:59:11.112267 29037 ts_tablet_manager.cc:1397] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a: Time spent bootstrapping tablet: real 0.040s user 0.013s sys 0.019s
I20250905 19:59:11.115428 29037 raft_consensus.cc:357] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:11.116382 29037 raft_consensus.cc:738] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Initialized, Role: LEARNER
I20250905 19:59:11.117132 29037 consensus_queue.cc:260] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:11.128530 29038 tablet_bootstrap.cc:492] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:11.131188 29037 ts_tablet_manager.cc:1428] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a: Time spent starting tablet: real 0.019s user 0.008s sys 0.000s
I20250905 19:59:11.132129 28749 raft_consensus.cc:1215] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250905 19:59:11.138831 28169 tablet_copy_service.cc:342] P 26b2f874b8e54f08b01a9ed00d5cce05: Request end of tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-0adebfdecbae47298f234b849fcea7fc received from {username='slave'} at 127.0.0.1:39586
I20250905 19:59:11.139326 28169 tablet_copy_service.cc:434] P 26b2f874b8e54f08b01a9ed00d5cce05: ending tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-0adebfdecbae47298f234b849fcea7fc on tablet 0adebfdecbae47298f234b849fcea7fc with peer 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:11.139003 29038 tablet_bootstrap.cc:492] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap complete.
I20250905 19:59:11.140534 29038 ts_tablet_manager.cc:1397] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a: Time spent bootstrapping tablet: real 0.047s user 0.025s sys 0.006s
I20250905 19:59:11.160341 29038 raft_consensus.cc:357] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:11.161161 29038 raft_consensus.cc:738] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Initialized, Role: LEARNER
I20250905 19:59:11.161770 29038 consensus_queue.cc:260] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:11.164508 29038 ts_tablet_manager.cc:1428] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a: Time spent starting tablet: real 0.024s user 0.005s sys 0.001s
I20250905 19:59:11.166392 28169 tablet_copy_service.cc:342] P 26b2f874b8e54f08b01a9ed00d5cce05: Request end of tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-c0889204295e4daa99d81e5b2a45ac1c received from {username='slave'} at 127.0.0.1:39586
I20250905 19:59:11.166873 28169 tablet_copy_service.cc:434] P 26b2f874b8e54f08b01a9ed00d5cce05: ending tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-c0889204295e4daa99d81e5b2a45ac1c on tablet c0889204295e4daa99d81e5b2a45ac1c with peer 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:11.272418 28749 raft_consensus.cc:1215] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250905 19:59:11.398051 28750 raft_consensus.cc:1215] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250905 19:59:11.490754 28750 raft_consensus.cc:1215] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250905 19:59:11.597146 29016 raft_consensus.cc:1062] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0: attempting to promote NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a to VOTER
I20250905 19:59:11.599668 29016 consensus_queue.cc:237] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:11.606055 28750 raft_consensus.cc:1273] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:11.606918 28159 raft_consensus.cc:1273] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:11.608407 28079 raft_consensus.cc:1273] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:11.608433 29016 consensus_queue.cc:1035] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20250905 19:59:11.611186 28947 consensus_queue.cc:1035] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:11.612962 29048 consensus_queue.cc:1035] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:11.646683 29017 raft_consensus.cc:2953] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:11.652228 28749 raft_consensus.cc:2953] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:11.674299 28159 raft_consensus.cc:2953] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:11.679755 27907 catalog_manager.cc:5582] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 19:59:11.698541 28079 raft_consensus.cc:2953] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:11.752224 28010 consensus_queue.cc:237] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:11.767360 28750 raft_consensus.cc:1273] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:11.767393 28079 raft_consensus.cc:1273] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:11.768924 28947 consensus_queue.cc:1035] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:59:11.770155 29048 consensus_queue.cc:1035] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:59:11.780952 29016 raft_consensus.cc:2953] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:11.785192 28947 raft_consensus.cc:1062] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0: attempting to promote NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a to VOTER
I20250905 19:59:11.805001 28947 consensus_queue.cc:237] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:11.813628 27895 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 62ccdb07048e4762b1b31c36b14610f5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20250905 19:59:11.825304 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 47000.8us
I20250905 19:59:11.831699 28749 raft_consensus.cc:2953] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:11.871934 27905 catalog_manager.cc:5582] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:11.885170 28990 raft_consensus.cc:1062] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05: attempting to promote NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a to VOTER
I20250905 19:59:11.888816 28990 consensus_queue.cc:237] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:11.921171 28145 tablet_service.cc:1515] Processing DeleteTablet for tablet 62ccdb07048e4762b1b31c36b14610f5 with delete_type TABLET_DATA_TOMBSTONED (TS 26b2f874b8e54f08b01a9ed00d5cce05 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33742
I20250905 19:59:11.936435 28079 raft_consensus.cc:1273] T 18ed409dabfc45b1843e7d8a316a047d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:11.938947 28150 raft_consensus.cc:1273] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:11.939818 28750 raft_consensus.cc:1273] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:11.941123 29061 consensus_queue.cc:1035] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:11.943382 29048 consensus_queue.cc:1035] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:11.952899 29016 consensus_queue.cc:1035] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:11.971558 29027 raft_consensus.cc:1062] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05: attempting to promote NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a to VOTER
I20250905 19:59:11.974561 29027 consensus_queue.cc:237] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:11.996114 29066 tablet_replica.cc:331] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05: stopping tablet replica
I20250905 19:59:11.997220 28082 raft_consensus.cc:2953] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:11.998953 28151 raft_consensus.cc:2953] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:11.997330 29060 raft_consensus.cc:2953] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:12.010735 29066 raft_consensus.cc:2241] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:12.010356 28750 raft_consensus.cc:2953] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:12.011584 29066 raft_consensus.cc:2270] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:12.022995 28079 raft_consensus.cc:2953] T 18ed409dabfc45b1843e7d8a316a047d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:12.040338 28004 raft_consensus.cc:1273] T 0adebfdecbae47298f234b849fcea7fc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:12.042457 29032 consensus_queue.cc:1035] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:12.061199 28751 raft_consensus.cc:1273] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:12.063946 29032 consensus_queue.cc:1035] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:12.065897 27905 catalog_manager.cc:5582] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 19:59:12.084192 28004 consensus_queue.cc:237] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:12.083225 29027 raft_consensus.cc:2953] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:12.092209 28006 raft_consensus.cc:2953] T 0adebfdecbae47298f234b849fcea7fc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:12.114629 28151 raft_consensus.cc:1273] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:12.116199 29048 consensus_queue.cc:1035] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 19:59:12.125880 29060 raft_consensus.cc:2953] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:12.141002 28151 raft_consensus.cc:2953] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:12.143352 28752 raft_consensus.cc:1273] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:12.145650 29032 consensus_queue.cc:1035] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:12.146549 28006 raft_consensus.cc:1273] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:12.148810 29032 consensus_queue.cc:1035] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250905 19:59:12.152580 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 116201us
I20250905 19:59:12.135004 27905 catalog_manager.cc:5582] T 0adebfdecbae47298f234b849fcea7fc P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:12.157279 29066 ts_tablet_manager.cc:1905] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 19:59:12.172834 28751 raft_consensus.cc:1273] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:12.180364 29062 consensus_queue.cc:1035] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 19:59:12.182943 27895 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 18ed409dabfc45b1843e7d8a316a047d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 19:59:12.186816 28751 raft_consensus.cc:2953] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:12.198710 29066 ts_tablet_manager.cc:1918] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250905 19:59:12.199226 29066 log.cc:1199] T 62ccdb07048e4762b1b31c36b14610f5 P 26b2f874b8e54f08b01a9ed00d5cce05: Deleting WAL directory at /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-2-root/wals/62ccdb07048e4762b1b31c36b14610f5
I20250905 19:59:12.201632 28079 raft_consensus.cc:1273] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:12.201318 28752 raft_consensus.cc:2953] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:12.203684 29027 consensus_queue.cc:1035] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:12.208298 27894 catalog_manager.cc:4928] TS 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): tablet 62ccdb07048e4762b1b31c36b14610f5 (table test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]) successfully deleted
I20250905 19:59:12.209862 28079 raft_consensus.cc:1273] T c0889204295e4daa99d81e5b2a45ac1c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:12.244055 29027 consensus_queue.cc:1035] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:12.257258 27905 catalog_manager.cc:5582] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:12.312386 29075 raft_consensus.cc:2953] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
W20250905 19:59:12.347079 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 59746.7us
I20250905 19:59:12.345554 28749 raft_consensus.cc:2953] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:12.371114 28072 tablet_service.cc:1515] Processing DeleteTablet for tablet 18ed409dabfc45b1843e7d8a316a047d with delete_type TABLET_DATA_TOMBSTONED (TS e4413ae5e0b64b52a6f360ac580239b9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59536
I20250905 19:59:12.373756 28079 raft_consensus.cc:2953] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:12.374264 28006 raft_consensus.cc:2953] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
W20250905 19:59:12.426860 29076 log.cc:927] Time spent T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20250905 19:59:12.443704 29081 tablet_replica.cc:331] T 18ed409dabfc45b1843e7d8a316a047d P e4413ae5e0b64b52a6f360ac580239b9: stopping tablet replica
I20250905 19:59:12.445032 29081 raft_consensus.cc:2241] T 18ed409dabfc45b1843e7d8a316a047d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:12.445885 29081 raft_consensus.cc:2270] T 18ed409dabfc45b1843e7d8a316a047d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:12.450371 27908 catalog_manager.cc:5582] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:12.489902 28150 consensus_queue.cc:237] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:12.495175 29081 ts_tablet_manager.cc:1905] T 18ed409dabfc45b1843e7d8a316a047d P e4413ae5e0b64b52a6f360ac580239b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 19:59:12.495640 28749 raft_consensus.cc:1273] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:12.497126 28990 consensus_queue.cc:1035] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 19:59:12.503250 28081 raft_consensus.cc:2953] T c0889204295e4daa99d81e5b2a45ac1c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:12.504974 29032 raft_consensus.cc:2953] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:12.512543 28751 raft_consensus.cc:2953] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:12.518271 29081 ts_tablet_manager.cc:1918] T 18ed409dabfc45b1843e7d8a316a047d P e4413ae5e0b64b52a6f360ac580239b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250905 19:59:12.518801 29081 log.cc:1199] T 18ed409dabfc45b1843e7d8a316a047d P e4413ae5e0b64b52a6f360ac580239b9: Deleting WAL directory at /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-1-root/wals/18ed409dabfc45b1843e7d8a316a047d
I20250905 19:59:12.526402 28002 raft_consensus.cc:1273] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:12.530905 29075 consensus_queue.cc:1035] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:59:12.532665 27906 catalog_manager.cc:5582] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:12.539849 28002 raft_consensus.cc:2953] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:12.542974 27894 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet c0889204295e4daa99d81e5b2a45ac1c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 19:59:12.554654 27892 catalog_manager.cc:4928] TS e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): tablet 18ed409dabfc45b1843e7d8a316a047d (table test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]) successfully deleted
I20250905 19:59:12.571480 28072 tablet_service.cc:1515] Processing DeleteTablet for tablet c0889204295e4daa99d81e5b2a45ac1c with delete_type TABLET_DATA_TOMBSTONED (TS e4413ae5e0b64b52a6f360ac580239b9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59536
I20250905 19:59:12.574959 29081 tablet_replica.cc:331] T c0889204295e4daa99d81e5b2a45ac1c P e4413ae5e0b64b52a6f360ac580239b9: stopping tablet replica
I20250905 19:59:12.608397 29081 raft_consensus.cc:2241] T c0889204295e4daa99d81e5b2a45ac1c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:12.609171 29081 raft_consensus.cc:2270] T c0889204295e4daa99d81e5b2a45ac1c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:12.612933 29081 ts_tablet_manager.cc:1905] T c0889204295e4daa99d81e5b2a45ac1c P e4413ae5e0b64b52a6f360ac580239b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 19:59:12.630347 29081 ts_tablet_manager.cc:1918] T c0889204295e4daa99d81e5b2a45ac1c P e4413ae5e0b64b52a6f360ac580239b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250905 19:59:12.630856 29081 log.cc:1199] T c0889204295e4daa99d81e5b2a45ac1c P e4413ae5e0b64b52a6f360ac580239b9: Deleting WAL directory at /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-1-root/wals/c0889204295e4daa99d81e5b2a45ac1c
I20250905 19:59:12.632519 27892 catalog_manager.cc:4928] TS e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): tablet c0889204295e4daa99d81e5b2a45ac1c (table test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]) successfully deleted
I20250905 19:59:12.828749 28150 consensus_queue.cc:237] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:12.833386 28751 raft_consensus.cc:1273] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:12.835050 28990 consensus_queue.cc:1035] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:59:12.837349 28081 raft_consensus.cc:1273] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:12.838723 28990 consensus_queue.cc:1035] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 19:59:12.851248 28990 raft_consensus.cc:2953] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:12.853240 28749 raft_consensus.cc:2953] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:12.867420 27894 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 0adebfdecbae47298f234b849fcea7fc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 19:59:12.873685 28081 raft_consensus.cc:2953] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:12.903213 27908 catalog_manager.cc:5582] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 19:59:12.926061 28000 tablet_service.cc:1515] Processing DeleteTablet for tablet 0adebfdecbae47298f234b849fcea7fc with delete_type TABLET_DATA_TOMBSTONED (TS cd764bfeebac4cc9ad851cd4a6a7d3b0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54144
I20250905 19:59:12.963128 29083 tablet_replica.cc:331] T 0adebfdecbae47298f234b849fcea7fc P cd764bfeebac4cc9ad851cd4a6a7d3b0: stopping tablet replica
I20250905 19:59:12.964421 29083 raft_consensus.cc:2241] T 0adebfdecbae47298f234b849fcea7fc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:12.965224 29083 raft_consensus.cc:2270] T 0adebfdecbae47298f234b849fcea7fc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:12.968900 29083 ts_tablet_manager.cc:1905] T 0adebfdecbae47298f234b849fcea7fc P cd764bfeebac4cc9ad851cd4a6a7d3b0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 19:59:12.984125 29083 ts_tablet_manager.cc:1918] T 0adebfdecbae47298f234b849fcea7fc P cd764bfeebac4cc9ad851cd4a6a7d3b0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250905 19:59:12.984671 29083 log.cc:1199] T 0adebfdecbae47298f234b849fcea7fc P cd764bfeebac4cc9ad851cd4a6a7d3b0: Deleting WAL directory at /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-0-root/wals/0adebfdecbae47298f234b849fcea7fc
I20250905 19:59:12.986653 27895 catalog_manager.cc:4928] TS cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): tablet 0adebfdecbae47298f234b849fcea7fc (table test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]) successfully deleted
I20250905 19:59:13.939625 28151 consensus_queue.cc:237] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:13.945353 28081 raft_consensus.cc:1273] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250905 19:59:13.945525 28123 consensus_peers.cc:489] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 -> Peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299): Couldn't send request to peer 7e9c9f591f05464f9b097bf003d14f4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6dbd54765afc49cb862b267d6663000c. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:13.945681 28002 raft_consensus.cc:1273] T 6dbd54765afc49cb862b267d6663000c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:13.946952 29032 consensus_queue.cc:1035] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:13.948256 29064 consensus_queue.cc:1035] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:13.970403 29032 raft_consensus.cc:2953] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:13.972976 28006 raft_consensus.cc:2953] T 6dbd54765afc49cb862b267d6663000c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:13.979954 28081 raft_consensus.cc:2953] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:14.012542 28002 consensus_queue.cc:237] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:14.019716 28159 raft_consensus.cc:1273] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250905 19:59:14.020272 27974 consensus_peers.cc:489] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 -> Peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299): Couldn't send request to peer 7e9c9f591f05464f9b097bf003d14f4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b2fd124328472aa3e3f62e0b327582. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:14.015573 27908 catalog_manager.cc:5582] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:14.022718 29085 consensus_queue.cc:1035] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:14.030405 28081 raft_consensus.cc:1273] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:14.037015 29048 consensus_queue.cc:1035] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:14.064114 29060 raft_consensus.cc:2953] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:14.079663 28150 raft_consensus.cc:2953] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:14.087630 28081 consensus_queue.cc:237] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:14.093917 27905 catalog_manager.cc:5582] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:14.094202 28084 raft_consensus.cc:2953] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:14.118376 28150 raft_consensus.cc:1273] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250905 19:59:14.119526 28049 consensus_peers.cc:489] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 -> Peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299): Couldn't send request to peer 7e9c9f591f05464f9b097bf003d14f4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2de38aacf1b4dccaeee999400fb6d75. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:14.120492 29079 consensus_queue.cc:1035] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:14.128122 28002 raft_consensus.cc:1273] T f2de38aacf1b4dccaeee999400fb6d75 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:14.131136 29051 consensus_queue.cc:1035] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:14.148442 29099 raft_consensus.cc:2953] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:14.150357 28151 raft_consensus.cc:2953] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:14.171289 28006 consensus_queue.cc:237] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:14.176812 27905 catalog_manager.cc:5582] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:14.179679 28150 raft_consensus.cc:1273] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:14.179857 28010 raft_consensus.cc:2953] T f2de38aacf1b4dccaeee999400fb6d75 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
W20250905 19:59:14.181759 27974 consensus_peers.cc:489] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 -> Peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299): Couldn't send request to peer 7e9c9f591f05464f9b097bf003d14f4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b13af7609cd04cf5b21745527b70074d. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:14.183837 28081 raft_consensus.cc:1273] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:14.185382 29060 consensus_queue.cc:1035] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:14.186246 29048 consensus_queue.cc:1035] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:14.202033 29048 raft_consensus.cc:2953] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:14.215668 28081 raft_consensus.cc:2953] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:14.232995 28151 raft_consensus.cc:2953] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:14.246093 27905 catalog_manager.cc:5582] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:14.395661 29108 ts_tablet_manager.cc:927] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a: Initiating tablet copy from peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:14.397711 29108 tablet_copy_client.cc:323] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Beginning tablet copy session from remote peer at address 127.26.58.131:36705
I20250905 19:59:14.399330 28169 tablet_copy_service.cc:140] P 26b2f874b8e54f08b01a9ed00d5cce05: Received BeginTabletCopySession request for tablet 6dbd54765afc49cb862b267d6663000c from peer 7e9c9f591f05464f9b097bf003d14f4a ({username='slave'} at 127.0.0.1:39586)
I20250905 19:59:14.399859 28169 tablet_copy_service.cc:161] P 26b2f874b8e54f08b01a9ed00d5cce05: Beginning new tablet copy session on tablet 6dbd54765afc49cb862b267d6663000c from peer 7e9c9f591f05464f9b097bf003d14f4a at {username='slave'} at 127.0.0.1:39586: session id = 7e9c9f591f05464f9b097bf003d14f4a-6dbd54765afc49cb862b267d6663000c
I20250905 19:59:14.407519 28169 tablet_copy_source_session.cc:215] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:14.411878 29108 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dbd54765afc49cb862b267d6663000c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:14.432420 29108 tablet_copy_client.cc:806] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:14.433364 29108 tablet_copy_client.cc:670] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:14.435155 29110 ts_tablet_manager.cc:927] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a: Initiating tablet copy from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421)
I20250905 19:59:14.437098 29110 tablet_copy_client.cc:323] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Beginning tablet copy session from remote peer at address 127.26.58.129:38421
I20250905 19:59:14.437850 29108 tablet_copy_client.cc:538] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 19:59:14.442395 28019 tablet_copy_service.cc:140] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Received BeginTabletCopySession request for tablet 05b2fd124328472aa3e3f62e0b327582 from peer 7e9c9f591f05464f9b097bf003d14f4a ({username='slave'} at 127.0.0.1:45844)
I20250905 19:59:14.443071 28019 tablet_copy_service.cc:161] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Beginning new tablet copy session on tablet 05b2fd124328472aa3e3f62e0b327582 from peer 7e9c9f591f05464f9b097bf003d14f4a at {username='slave'} at 127.0.0.1:45844: session id = 7e9c9f591f05464f9b097bf003d14f4a-05b2fd124328472aa3e3f62e0b327582
I20250905 19:59:14.449028 28019 tablet_copy_source_session.cc:215] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:14.456632 29108 tablet_bootstrap.cc:492] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap starting.
I20250905 19:59:14.462218 29110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05b2fd124328472aa3e3f62e0b327582. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:14.486011 29110 tablet_copy_client.cc:806] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:14.486734 29110 tablet_copy_client.cc:670] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:14.490371 29110 tablet_copy_client.cc:538] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 19:59:14.491268 29108 tablet_bootstrap.cc:492] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:14.492563 29108 tablet_bootstrap.cc:492] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap complete.
I20250905 19:59:14.494763 29108 ts_tablet_manager.cc:1397] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a: Time spent bootstrapping tablet: real 0.038s user 0.027s sys 0.009s
I20250905 19:59:14.497745 29108 raft_consensus.cc:357] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:14.498836 29108 raft_consensus.cc:738] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Initialized, Role: LEARNER
I20250905 19:59:14.499572 29108 consensus_queue.cc:260] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:14.501870 29110 tablet_bootstrap.cc:492] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap starting.
I20250905 19:59:14.519443 29108 ts_tablet_manager.cc:1428] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a: Time spent starting tablet: real 0.024s user 0.021s sys 0.000s
I20250905 19:59:14.521279 28169 tablet_copy_service.cc:342] P 26b2f874b8e54f08b01a9ed00d5cce05: Request end of tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-6dbd54765afc49cb862b267d6663000c received from {username='slave'} at 127.0.0.1:39586
I20250905 19:59:14.521759 28169 tablet_copy_service.cc:434] P 26b2f874b8e54f08b01a9ed00d5cce05: ending tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-6dbd54765afc49cb862b267d6663000c on tablet 6dbd54765afc49cb862b267d6663000c with peer 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:14.525440 29110 tablet_bootstrap.cc:492] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:14.526469 29110 tablet_bootstrap.cc:492] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap complete.
I20250905 19:59:14.527244 29110 ts_tablet_manager.cc:1397] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent bootstrapping tablet: real 0.026s user 0.015s sys 0.008s
I20250905 19:59:14.530290 29110 raft_consensus.cc:357] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:14.531177 29110 raft_consensus.cc:738] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Initialized, Role: LEARNER
I20250905 19:59:14.531822 29110 consensus_queue.cc:260] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:14.534718 29110 ts_tablet_manager.cc:1428] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250905 19:59:14.536789 28019 tablet_copy_service.cc:342] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Request end of tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-05b2fd124328472aa3e3f62e0b327582 received from {username='slave'} at 127.0.0.1:45844
I20250905 19:59:14.537261 28019 tablet_copy_service.cc:434] P cd764bfeebac4cc9ad851cd4a6a7d3b0: ending tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-05b2fd124328472aa3e3f62e0b327582 on tablet 05b2fd124328472aa3e3f62e0b327582 with peer 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:14.540813 29108 ts_tablet_manager.cc:927] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a: Initiating tablet copy from peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:14.542724 29108 tablet_copy_client.cc:323] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Beginning tablet copy session from remote peer at address 127.26.58.130:34193
I20250905 19:59:14.544318 28095 tablet_copy_service.cc:140] P e4413ae5e0b64b52a6f360ac580239b9: Received BeginTabletCopySession request for tablet f2de38aacf1b4dccaeee999400fb6d75 from peer 7e9c9f591f05464f9b097bf003d14f4a ({username='slave'} at 127.0.0.1:50660)
I20250905 19:59:14.544895 28095 tablet_copy_service.cc:161] P e4413ae5e0b64b52a6f360ac580239b9: Beginning new tablet copy session on tablet f2de38aacf1b4dccaeee999400fb6d75 from peer 7e9c9f591f05464f9b097bf003d14f4a at {username='slave'} at 127.0.0.1:50660: session id = 7e9c9f591f05464f9b097bf003d14f4a-f2de38aacf1b4dccaeee999400fb6d75
I20250905 19:59:14.559465 28095 tablet_copy_source_session.cc:215] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:14.562762 29108 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2de38aacf1b4dccaeee999400fb6d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:14.579953 29108 tablet_copy_client.cc:806] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:14.580562 29108 tablet_copy_client.cc:670] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:14.586969 29108 tablet_copy_client.cc:538] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 19:59:14.590973 29110 ts_tablet_manager.cc:927] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a: Initiating tablet copy from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421)
I20250905 19:59:14.592736 29110 tablet_copy_client.cc:323] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Beginning tablet copy session from remote peer at address 127.26.58.129:38421
I20250905 19:59:14.595278 28019 tablet_copy_service.cc:140] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Received BeginTabletCopySession request for tablet b13af7609cd04cf5b21745527b70074d from peer 7e9c9f591f05464f9b097bf003d14f4a ({username='slave'} at 127.0.0.1:45844)
I20250905 19:59:14.595857 28019 tablet_copy_service.cc:161] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Beginning new tablet copy session on tablet b13af7609cd04cf5b21745527b70074d from peer 7e9c9f591f05464f9b097bf003d14f4a at {username='slave'} at 127.0.0.1:45844: session id = 7e9c9f591f05464f9b097bf003d14f4a-b13af7609cd04cf5b21745527b70074d
I20250905 19:59:14.597103 29108 tablet_bootstrap.cc:492] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap starting.
I20250905 19:59:14.619294 28019 tablet_copy_source_session.cc:215] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:14.619302 29108 tablet_bootstrap.cc:492] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:14.620631 29108 tablet_bootstrap.cc:492] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap complete.
I20250905 19:59:14.621335 29108 ts_tablet_manager.cc:1397] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent bootstrapping tablet: real 0.024s user 0.024s sys 0.000s
I20250905 19:59:14.623175 29110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b13af7609cd04cf5b21745527b70074d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:14.624285 29108 raft_consensus.cc:357] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:14.625298 29108 raft_consensus.cc:738] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Initialized, Role: LEARNER
I20250905 19:59:14.625969 29108 consensus_queue.cc:260] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:14.635998 29108 ts_tablet_manager.cc:1428] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent starting tablet: real 0.014s user 0.009s sys 0.000s
I20250905 19:59:14.640075 28095 tablet_copy_service.cc:342] P e4413ae5e0b64b52a6f360ac580239b9: Request end of tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-f2de38aacf1b4dccaeee999400fb6d75 received from {username='slave'} at 127.0.0.1:50660
I20250905 19:59:14.640564 28095 tablet_copy_service.cc:434] P e4413ae5e0b64b52a6f360ac580239b9: ending tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-f2de38aacf1b4dccaeee999400fb6d75 on tablet f2de38aacf1b4dccaeee999400fb6d75 with peer 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:14.655154 29110 tablet_copy_client.cc:806] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:14.655802 29110 tablet_copy_client.cc:670] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:14.659166 29110 tablet_copy_client.cc:538] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 19:59:14.667936 29110 tablet_bootstrap.cc:492] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap starting.
I20250905 19:59:14.685861 29110 tablet_bootstrap.cc:492] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:14.686800 29110 tablet_bootstrap.cc:492] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap complete.
I20250905 19:59:14.687402 29110 ts_tablet_manager.cc:1397] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a: Time spent bootstrapping tablet: real 0.020s user 0.019s sys 0.000s
I20250905 19:59:14.690070 29110 raft_consensus.cc:357] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:14.690958 29110 raft_consensus.cc:738] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Initialized, Role: LEARNER
I20250905 19:59:14.691618 29110 consensus_queue.cc:260] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:14.694604 29110 ts_tablet_manager.cc:1428] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250905 19:59:14.696192 28019 tablet_copy_service.cc:342] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Request end of tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-b13af7609cd04cf5b21745527b70074d received from {username='slave'} at 127.0.0.1:45844
I20250905 19:59:14.696641 28019 tablet_copy_service.cc:434] P cd764bfeebac4cc9ad851cd4a6a7d3b0: ending tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-b13af7609cd04cf5b21745527b70074d on tablet b13af7609cd04cf5b21745527b70074d with peer 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:14.836126 28749 raft_consensus.cc:1215] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250905 19:59:14.850149 28749 raft_consensus.cc:1215] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250905 19:59:14.970685 28751 raft_consensus.cc:1215] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250905 19:59:15.066155 28751 raft_consensus.cc:1215] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20250905 19:59:16.166497 27974 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.758s user 0.008s sys 0.015s
W20250905 19:59:16.166852 27974 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.758s user 0.008s sys 0.015s
I20250905 19:59:16.185153 29130 raft_consensus.cc:491] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:16.186046 29130 raft_consensus.cc:513] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:16.189610 29130 leader_election.cc:290] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:16.195652 29048 raft_consensus.cc:1062] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0: attempting to promote NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a to VOTER
I20250905 19:59:16.210714 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:16.216105 29048 consensus_queue.cc:237] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:16.231784 29084 raft_consensus.cc:1062] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9: attempting to promote NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a to VOTER
I20250905 19:59:16.232432 29079 raft_consensus.cc:491] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:16.232856 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:16.233021 29079 raft_consensus.cc:513] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:16.235353 29079 leader_election.cc:290] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:16.234743 29084 consensus_queue.cc:237] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:16.253036 28715 leader_election.cc:304] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:16.254184 29130 raft_consensus.cc:2747] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.286151 29125 raft_consensus.cc:491] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:16.286917 29125 raft_consensus.cc:513] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.289083 29125 leader_election.cc:290] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:16.291273 29125 raft_consensus.cc:491] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:16.291868 29125 raft_consensus.cc:513] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.294586 29104 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:16.294915 29125 leader_election.cc:290] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:16.295393 29104 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:16.298228 29104 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:16.303702 28749 raft_consensus.cc:1273] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:16.306069 29085 raft_consensus.cc:491] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:16.307044 29085 raft_consensus.cc:513] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.310453 29085 leader_election.cc:290] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:16.311939 29085 raft_consensus.cc:491] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:16.312546 29085 raft_consensus.cc:513] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.315140 29085 leader_election.cc:290] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:16.317078 27939 debug-util.cc:398] Leaking SignalData structure 0x7b08001c4bc0 after lost signal to thread 26861
I20250905 19:59:16.322443 29136 consensus_queue.cc:1035] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20250905 19:59:16.324357 27939 debug-util.cc:398] Leaking SignalData structure 0x7b08001e8e40 after lost signal to thread 27942
W20250905 19:59:16.325558 27939 debug-util.cc:398] Leaking SignalData structure 0x7b080030da80 after lost signal to thread 28034
W20250905 19:59:16.326838 27939 debug-util.cc:398] Leaking SignalData structure 0x7b08002ef400 after lost signal to thread 28109
W20250905 19:59:16.327893 27939 debug-util.cc:398] Leaking SignalData structure 0x7b0800226460 after lost signal to thread 28183
W20250905 19:59:16.328996 27939 debug-util.cc:398] Leaking SignalData structure 0x7b080015bb40 after lost signal to thread 28776
I20250905 19:59:16.330591 29104 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:16.331236 29104 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.333719 29104 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:16.342147 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:16.343580 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:16.345120 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:16.350970 29128 raft_consensus.cc:491] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:16.351589 29128 raft_consensus.cc:513] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.353852 29128 leader_election.cc:290] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:16.355206 29119 raft_consensus.cc:491] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:16.355640 29048 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:16.356091 29120 raft_consensus.cc:491] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:16.355793 29119 raft_consensus.cc:513] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.356191 29048 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.356637 29120 raft_consensus.cc:513] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.358095 29119 leader_election.cc:290] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:16.358500 29048 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:16.351579 29089 raft_consensus.cc:491] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:16.359483 29120 leader_election.cc:290] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:16.359854 29089 raft_consensus.cc:513] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:16.362301 29089 leader_election.cc:290] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 19:59:16.360188 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 133656us
I20250905 19:59:16.363395 29128 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:16.364166 29128 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.366076 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:16.366479 29128 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:16.367174 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:16.367832 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:16.367962 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 166056us
I20250905 19:59:16.368377 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:16.371588 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:16.372515 29048 raft_consensus.cc:491] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:16.373926 29126 raft_consensus.cc:491] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:16.374624 29126 raft_consensus.cc:513] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.377326 28751 raft_consensus.cc:1273] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:16.375912 29129 raft_consensus.cc:491] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:16.373657 29048 raft_consensus.cc:513] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.376397 29089 raft_consensus.cc:491] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:16.378840 29129 raft_consensus.cc:513] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.379117 29089 raft_consensus.cc:513] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:16.380144 29122 raft_consensus.cc:491] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:16.380735 29122 raft_consensus.cc:513] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.381166 29129 leader_election.cc:290] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:16.381284 29089 leader_election.cc:290] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:16.378278 29126 leader_election.cc:290] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:16.383136 29122 leader_election.cc:290] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:16.383363 29048 leader_election.cc:290] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:16.384114 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:16.385398 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:16.385771 28081 raft_consensus.cc:1273] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:16.389318 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:16.389396 29130 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:16.390036 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:16.390659 29130 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:16.392721 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:16.393512 29130 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:16.394596 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:16.395618 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:16.388859 29122 raft_consensus.cc:1062] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0: attempting to promote NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a to VOTER
I20250905 19:59:16.397648 29089 raft_consensus.cc:491] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:16.399752 29048 consensus_queue.cc:1035] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:16.401328 29122 consensus_queue.cc:237] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:16.402586 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:16.407261 28010 raft_consensus.cc:2466] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 1.
I20250905 19:59:16.403528 29085 consensus_queue.cc:1035] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:16.409488 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:16.399451 29128 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:16.416409 29131 raft_consensus.cc:491] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:16.415055 29128 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.419616 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:16.421128 28010 raft_consensus.cc:1273] T f2de38aacf1b4dccaeee999400fb6d75 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:16.408152 29089 raft_consensus.cc:513] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.412786 28718 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:16.421909 29127 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:16.427227 29127 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.428402 29130 raft_consensus.cc:2802] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:16.428870 29130 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:16.429234 29130 raft_consensus.cc:3058] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:16.417308 29131 raft_consensus.cc:513] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.433427 29131 leader_election.cc:290] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:16.435639 29139 raft_consensus.cc:491] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:16.435492 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:16.436287 29139 raft_consensus.cc:513] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.439994 29089 leader_election.cc:290] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:16.441758 29128 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:16.443893 29127 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:16.445092 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:16.451045 29139 leader_election.cc:290] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:16.451550 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:16.451715 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:16.451195 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:16.456730 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:16.457336 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:16.457968 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:16.458205 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:16.459074 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:16.459174 29123 consensus_queue.cc:1035] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20250905 19:59:16.459825 28155 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:59:16.459745 29141 raft_consensus.cc:491] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:16.461905 29133 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:16.455770 29089 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:16.462671 29133 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.464418 29129 raft_consensus.cc:491] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:16.465022 29129 raft_consensus.cc:513] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.463378 29089 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.465824 29085 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:16.467242 29129 leader_election.cc:290] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:16.467636 29089 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:16.467090 29085 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:16.468356 29133 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:16.476605 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:16.478566 28151 raft_consensus.cc:1273] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:16.479456 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:16.480854 29085 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:16.481158 29130 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:16.482046 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:16.482333 28751 raft_consensus.cc:1238] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:16.483095 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:16.487519 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:16.487792 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:16.488354 28150 raft_consensus.cc:2466] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:59:16.488739 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:16.488795 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:16.498039 28752 raft_consensus.cc:1273] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:16.499607 29133 consensus_queue.cc:1035] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:16.502233 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:16.503624 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:16.460680 29141 raft_consensus.cc:513] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:16.509680 29139 raft_consensus.cc:491] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:16.510298 29139 raft_consensus.cc:513] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.511358 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:16.511940 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:16.512872 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:16.513528 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:16.515429 29089 consensus_queue.cc:1046] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250905 19:59:16.512326 29128 raft_consensus.cc:491] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:16.512399 29139 leader_election.cc:290] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:16.517381 29128 raft_consensus.cc:513] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.519384 29130 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:16.519495 29128 leader_election.cc:290] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:16.520364 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:16.526250 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:16.527477 29143 raft_consensus.cc:491] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:16.538348 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:16.539302 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:16.529630 29139 raft_consensus.cc:3053] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:16.530130 29132 raft_consensus.cc:1062] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05: attempting to promote NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a to VOTER
I20250905 19:59:16.541137 29139 raft_consensus.cc:738] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:16.560309 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:16.538975 29143 raft_consensus.cc:513] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.540279 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:16.564303 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:16.564425 29089 raft_consensus.cc:491] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:16.568240 29139 consensus_queue.cc:260] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:16.579967 29089 raft_consensus.cc:513] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.580338 29139 raft_consensus.cc:3058] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:16.569638 28152 raft_consensus.cc:1273] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:16.570407 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:16.571884 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:16.572554 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:16.573313 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:16.573654 29141 leader_election.cc:290] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:16.574571 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:16.574826 29143 leader_election.cc:290] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:16.582871 29089 leader_election.cc:290] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:16.575151 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:16.574842 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:16.575574 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:16.575727 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:16.578389 28080 raft_consensus.cc:1273] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:16.578779 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:16.572889 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:16.579030 29132 consensus_queue.cc:237] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:16.591028 28125 leader_election.cc:304] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:16.592082 29143 raft_consensus.cc:2747] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.593328 28125 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:16.594377 29089 raft_consensus.cc:2747] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.593422 29131 raft_consensus.cc:2953] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:16.595585 28125 leader_election.cc:304] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:16.596328 29089 raft_consensus.cc:2747] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.596716 28125 leader_election.cc:304] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:16.599484 28752 raft_consensus.cc:1273] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:16.603561 29132 raft_consensus.cc:2747] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.603783 29136 raft_consensus.cc:2953] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:16.607290 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:16.608997 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:16.610916 28080 raft_consensus.cc:1273] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:16.613099 29139 consensus_queue.cc:1035] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:16.614621 29143 raft_consensus.cc:491] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:16.615211 29143 raft_consensus.cc:513] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.616091 29139 consensus_queue.cc:1035] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:16.617233 29143 leader_election.cc:290] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:16.618135 28125 leader_election.cc:304] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:16.619024 29089 raft_consensus.cc:2747] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.620324 28125 leader_election.cc:304] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:16.621696 28749 raft_consensus.cc:2953] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:16.623603 29132 raft_consensus.cc:2747] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.624878 29132 raft_consensus.cc:491] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:16.625356 29132 raft_consensus.cc:513] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.627319 29132 leader_election.cc:290] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:16.627605 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:16.632889 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:16.631395 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:16.634999 28159 raft_consensus.cc:1273] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20250905 19:59:16.679116 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 73459.4us
W20250905 19:59:16.683707 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:16.696357 28125 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:16.682075 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:16.703330 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:16.707929 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:16.710091 28749 raft_consensus.cc:2953] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:16.717209 29089 raft_consensus.cc:2747] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.731500 28125 leader_election.cc:304] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:16.747228 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:16.745973 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:16.748210 29048 consensus_queue.cc:1035] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20250905 19:59:16.754277 29141 raft_consensus.cc:2747] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.770514 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:16.772634 28003 raft_consensus.cc:1273] T 6dbd54765afc49cb862b267d6663000c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:16.776101 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:16.775530 27906 catalog_manager.cc:5582] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
W20250905 19:59:16.779006 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 47820.5us
I20250905 19:59:16.779677 28718 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:16.780114 28125 leader_election.cc:304] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:16.780719 29130 raft_consensus.cc:2747] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250905 19:59:16.781399 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 54299.3us
I20250905 19:59:16.782397 27975 leader_election.cc:304] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:16.786739 27908 catalog_manager.cc:5582] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 19:59:16.797194 29139 raft_consensus.cc:2747] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.797739 27975 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:16.795941 29105 raft_consensus.cc:2747] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.802675 28125 leader_election.cc:304] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:16.806023 29139 raft_consensus.cc:2747] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.808185 28125 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:16.809417 29139 raft_consensus.cc:2747] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.811089 29105 raft_consensus.cc:2747] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.811694 28125 leader_election.cc:304] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:16.811905 27975 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:16.812750 29139 raft_consensus.cc:2747] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.813125 29105 raft_consensus.cc:2802] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:16.813639 29105 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:16.814143 29105 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:16.814543 28125 leader_election.cc:304] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:16.808249 28050 leader_election.cc:304] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:16.815856 29125 raft_consensus.cc:2747] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.814008 27975 leader_election.cc:304] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:16.818858 29085 raft_consensus.cc:2747] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.821282 29085 consensus_queue.cc:1035] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:16.822252 27975 leader_election.cc:304] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:16.823807 29141 raft_consensus.cc:2747] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.824214 29119 raft_consensus.cc:2747] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.826103 27975 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:16.827199 29119 raft_consensus.cc:2802] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:16.827596 29119 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:16.828080 29119 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:16.824878 29141 raft_consensus.cc:2953] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:16.826124 28125 leader_election.cc:304] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:16.828459 28081 raft_consensus.cc:2953] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:16.830765 29143 raft_consensus.cc:2747] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.831334 27975 leader_election.cc:304] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:16.832571 28125 leader_election.cc:304] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:16.835348 29085 raft_consensus.cc:2747] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.838570 29143 raft_consensus.cc:2747] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.841845 29105 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.857865 28050 leader_election.cc:304] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:16.859714 28050 leader_election.cc:304] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:16.860484 29125 raft_consensus.cc:2747] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.861166 27975 leader_election.cc:304] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:16.863390 29048 raft_consensus.cc:2747] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.861984 29105 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:16.871398 28125 leader_election.cc:304] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:16.879712 29125 consensus_queue.cc:1035] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:16.870257 28752 raft_consensus.cc:2953] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:16.873912 29119 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:16.880959 27975 leader_election.cc:304] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:16.881294 29143 raft_consensus.cc:2747] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.881477 28125 leader_election.cc:304] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:16.886312 29119 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:16.886987 29119 raft_consensus.cc:2747] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.887548 29143 raft_consensus.cc:2747] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.887624 28125 leader_election.cc:304] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:16.888667 29143 raft_consensus.cc:2747] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.889531 28125 leader_election.cc:304] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:16.889348 29089 consensus_queue.cc:1035] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:16.892915 27975 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:16.879031 29121 raft_consensus.cc:2747] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.900887 27907 catalog_manager.cc:5582] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 19:59:16.905961 29119 raft_consensus.cc:2747] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.908042 27975 leader_election.cc:304] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:16.908960 29119 consensus_queue.cc:1035] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:16.911216 29119 raft_consensus.cc:2747] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.912907 27975 leader_election.cc:304] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:16.915504 29119 raft_consensus.cc:2747] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.914940 28159 raft_consensus.cc:2953] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:16.918090 29089 raft_consensus.cc:2747] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:16.918236 28002 raft_consensus.cc:2953] T f2de38aacf1b4dccaeee999400fb6d75 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:16.950388 28080 raft_consensus.cc:2953] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:17.005808 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:17.006335 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:17.019812 28159 consensus_queue.cc:237] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:17.026662 28752 raft_consensus.cc:1273] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:17.028278 29141 consensus_queue.cc:1035] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 19:59:17.029134 28004 raft_consensus.cc:2953] T 6dbd54765afc49cb862b267d6663000c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:17.033771 28003 raft_consensus.cc:1238] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:17.036311 29089 raft_consensus.cc:2953] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:17.043884 28752 raft_consensus.cc:2953] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:17.063753 28080 raft_consensus.cc:1273] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:17.072014 29143 consensus_queue.cc:1035] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:59:17.080217 28004 raft_consensus.cc:1238] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:17.088454 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:17.088276 27908 catalog_manager.cc:5582] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:17.093703 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:17.105855 27894 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 6dbd54765afc49cb862b267d6663000c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 19:59:17.115365 28152 raft_consensus.cc:2953] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:17.124701 28000 tablet_service.cc:1515] Processing DeleteTablet for tablet 6dbd54765afc49cb862b267d6663000c with delete_type TABLET_DATA_TOMBSTONED (TS cd764bfeebac4cc9ad851cd4a6a7d3b0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54144
I20250905 19:59:17.138993 27975 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:17.146224 29119 raft_consensus.cc:2747] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 19:59:17.150820 27975 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:17.155694 29105 raft_consensus.cc:2747] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 19:59:17.152383 29085 raft_consensus.cc:2953] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:17.158347 29125 consensus_queue.cc:1046] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:17.160019 29123 raft_consensus.cc:3053] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:17.160456 29123 raft_consensus.cc:738] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:17.161405 29123 consensus_queue.cc:260] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:17.163496 29123 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:17.166141 28080 raft_consensus.cc:2953] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:17.173542 28151 raft_consensus.cc:2953] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:17.193902 29084 consensus_queue.cc:1046] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:17.195029 29125 raft_consensus.cc:3053] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:17.195482 29125 raft_consensus.cc:738] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:17.196398 29125 consensus_queue.cc:260] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:17.197926 29125 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:17.198887 28079 raft_consensus.cc:2953] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:17.214390 29163 tablet_replica.cc:331] T 6dbd54765afc49cb862b267d6663000c P cd764bfeebac4cc9ad851cd4a6a7d3b0: stopping tablet replica
I20250905 19:59:17.236383 28749 raft_consensus.cc:2953] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:17.254979 29163 raft_consensus.cc:2241] T 6dbd54765afc49cb862b267d6663000c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:17.255784 29163 raft_consensus.cc:2270] T 6dbd54765afc49cb862b267d6663000c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:17.260951 29163 ts_tablet_manager.cc:1905] T 6dbd54765afc49cb862b267d6663000c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 19:59:17.264966 27908 catalog_manager.cc:5582] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 19:59:17.290402 29163 ts_tablet_manager.cc:1918] T 6dbd54765afc49cb862b267d6663000c P cd764bfeebac4cc9ad851cd4a6a7d3b0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250905 19:59:17.291162 29163 log.cc:1199] T 6dbd54765afc49cb862b267d6663000c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Deleting WAL directory at /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-0-root/wals/6dbd54765afc49cb862b267d6663000c
I20250905 19:59:17.308549 27895 catalog_manager.cc:4928] TS cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): tablet 6dbd54765afc49cb862b267d6663000c (table test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]) successfully deleted
I20250905 19:59:17.314945 28009 consensus_queue.cc:237] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:17.322268 28752 raft_consensus.cc:1273] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:17.323179 28079 raft_consensus.cc:1273] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:17.324792 29119 consensus_queue.cc:1035] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:59:17.332397 29085 raft_consensus.cc:2953] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:17.335021 28079 raft_consensus.cc:2953] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:17.336591 29048 consensus_queue.cc:1035] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:59:17.342237 28752 raft_consensus.cc:2953] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:17.356213 27895 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 05b2fd124328472aa3e3f62e0b327582 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 19:59:17.359494 28002 consensus_queue.cc:237] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:17.366317 28152 raft_consensus.cc:1273] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:17.367774 29085 consensus_queue.cc:1035] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:59:17.367718 28752 raft_consensus.cc:1273] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:17.375413 29119 consensus_queue.cc:1035] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:59:17.380117 29133 raft_consensus.cc:2953] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:17.385149 27908 catalog_manager.cc:5582] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:17.397640 28152 raft_consensus.cc:2953] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:17.402819 27895 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet b13af7609cd04cf5b21745527b70074d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 19:59:17.407042 28752 raft_consensus.cc:2953] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:17.419086 28145 tablet_service.cc:1515] Processing DeleteTablet for tablet 05b2fd124328472aa3e3f62e0b327582 with delete_type TABLET_DATA_TOMBSTONED (TS 26b2f874b8e54f08b01a9ed00d5cce05 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33742
I20250905 19:59:17.427939 29164 tablet_replica.cc:331] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05: stopping tablet replica
I20250905 19:59:17.440330 27905 catalog_manager.cc:5582] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:17.442623 29164 raft_consensus.cc:2241] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:17.443401 29164 raft_consensus.cc:2270] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:17.446787 29164 ts_tablet_manager.cc:1905] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 19:59:17.467483 28080 consensus_queue.cc:237] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:17.481145 28072 tablet_service.cc:1515] Processing DeleteTablet for tablet b13af7609cd04cf5b21745527b70074d with delete_type TABLET_DATA_TOMBSTONED (TS e4413ae5e0b64b52a6f360ac580239b9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59536
I20250905 19:59:17.495673 28151 raft_consensus.cc:1273] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:17.495836 28749 raft_consensus.cc:1273] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:17.498843 29160 consensus_queue.cc:1035] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 19:59:17.511250 29125 consensus_queue.cc:1035] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:59:17.522627 29125 raft_consensus.cc:2953] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:17.512220 29164 ts_tablet_manager.cc:1918] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250905 19:59:17.568432 28151 raft_consensus.cc:2953] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:17.541678 29164 log.cc:1199] T 05b2fd124328472aa3e3f62e0b327582 P 26b2f874b8e54f08b01a9ed00d5cce05: Deleting WAL directory at /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-2-root/wals/05b2fd124328472aa3e3f62e0b327582
I20250905 19:59:17.575145 27894 catalog_manager.cc:4928] TS 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): tablet 05b2fd124328472aa3e3f62e0b327582 (table test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]) successfully deleted
I20250905 19:59:17.577936 29165 tablet_replica.cc:331] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9: stopping tablet replica
I20250905 19:59:17.591423 29165 raft_consensus.cc:2241] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:17.592356 29165 raft_consensus.cc:2270] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:17.595613 28749 raft_consensus.cc:2953] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:17.597585 27905 catalog_manager.cc:5582] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:17.624048 27892 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet f2de38aacf1b4dccaeee999400fb6d75 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 19:59:17.636507 29165 ts_tablet_manager.cc:1905] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 19:59:17.638535 28000 tablet_service.cc:1515] Processing DeleteTablet for tablet f2de38aacf1b4dccaeee999400fb6d75 with delete_type TABLET_DATA_TOMBSTONED (TS cd764bfeebac4cc9ad851cd4a6a7d3b0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54144
I20250905 19:59:17.639554 29163 tablet_replica.cc:331] T f2de38aacf1b4dccaeee999400fb6d75 P cd764bfeebac4cc9ad851cd4a6a7d3b0: stopping tablet replica
I20250905 19:59:17.640648 29163 raft_consensus.cc:2241] T f2de38aacf1b4dccaeee999400fb6d75 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:17.641433 29163 raft_consensus.cc:2270] T f2de38aacf1b4dccaeee999400fb6d75 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:17.647208 29163 ts_tablet_manager.cc:1905] T f2de38aacf1b4dccaeee999400fb6d75 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 19:59:17.670382 29165 ts_tablet_manager.cc:1918] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250905 19:59:17.671463 29165 log.cc:1199] T b13af7609cd04cf5b21745527b70074d P e4413ae5e0b64b52a6f360ac580239b9: Deleting WAL directory at /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-1-root/wals/b13af7609cd04cf5b21745527b70074d
I20250905 19:59:17.674369 27892 catalog_manager.cc:4928] TS e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): tablet b13af7609cd04cf5b21745527b70074d (table test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]) successfully deleted
I20250905 19:59:17.676939 29163 ts_tablet_manager.cc:1918] T f2de38aacf1b4dccaeee999400fb6d75 P cd764bfeebac4cc9ad851cd4a6a7d3b0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250905 19:59:17.677527 29163 log.cc:1199] T f2de38aacf1b4dccaeee999400fb6d75 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Deleting WAL directory at /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-0-root/wals/f2de38aacf1b4dccaeee999400fb6d75
I20250905 19:59:17.679076 27895 catalog_manager.cc:4928] TS cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): tablet f2de38aacf1b4dccaeee999400fb6d75 (table test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]) successfully deleted
I20250905 19:59:18.144457 29139 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:18.145002 29139 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:18.147250 29139 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:18.148130 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:18.148849 28002 raft_consensus.cc:2466] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 19:59:18.148860 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:18.149715 28749 raft_consensus.cc:2466] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:18.149942 28125 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:18.150813 29139 raft_consensus.cc:2802] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:18.151142 29139 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:18.151423 29139 raft_consensus.cc:3058] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:18.156886 29139 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:18.158691 29139 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:18.159415 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:18.159811 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:18.160046 28002 raft_consensus.cc:3058] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 3
I20250905 19:59:18.160390 28749 raft_consensus.cc:3058] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:18.167178 28749 raft_consensus.cc:2466] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:18.168921 28123 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 19:59:18.170939 29089 raft_consensus.cc:2802] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:18.171619 29089 raft_consensus.cc:695] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:18.172075 28002 raft_consensus.cc:2466] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:18.172796 29089 consensus_queue.cc:237] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:18.196096 27906 catalog_manager.cc:5582] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:18.475630 29135 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:18.476094 29135 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:18.478173 29135 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:18.478992 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:18.479262 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:18.479624 28009 raft_consensus.cc:2466] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
I20250905 19:59:18.480017 28151 raft_consensus.cc:2466] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:59:18.480643 28051 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:18.481369 29135 raft_consensus.cc:2802] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:18.481678 29135 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:18.481938 29135 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:18.487499 29135 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:18.489650 29135 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:18.490069 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:18.490834 28009 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:18.491111 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:18.493005 28151 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 3
I20250905 19:59:18.499599 28009 raft_consensus.cc:2466] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:18.500963 28051 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:18.502138 29135 raft_consensus.cc:2802] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:18.503473 29135 raft_consensus.cc:695] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:18.504528 29135 consensus_queue.cc:237] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:18.505676 28151 raft_consensus.cc:2466] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:18.517659 27906 catalog_manager.cc:5582] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:18.654037 29089 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:18.654778 29089 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:18.658941 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:18.659252 29089 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:18.659938 28009 raft_consensus.cc:2391] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 2: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:18.660691 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:18.661404 28079 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:18.664024 28122 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:18.665252 29089 raft_consensus.cc:2802] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:18.665727 29089 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:18.666177 29089 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:18.676939 29089 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:18.679373 29089 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:18.679970 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:18.679988 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:18.680907 28009 raft_consensus.cc:2391] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 2: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:18.687604 28079 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:18.690202 28122 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:18.692209 29089 raft_consensus.cc:2802] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:18.693456 29089 raft_consensus.cc:695] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:18.695225 29089 consensus_queue.cc:237] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:18.726754 27906 catalog_manager.cc:5582] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 1 to 2, leader changed from e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 2 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20250905 19:59:18.747270 28079 consensus_queue.cc:237] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:18.758127 28009 raft_consensus.cc:1273] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250905 19:59:18.767879 29139 consensus_queue.cc:1035] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250905 19:59:18.781729 28749 raft_consensus.cc:1273] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250905 19:59:18.790953 29173 consensus_queue.cc:1035] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250905 19:59:18.832201 29133 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:18.832784 29133 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:18.842703 29133 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:18.845372 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:18.849972 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:18.850682 28080 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:18.852103 27973 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:18.864501 29104 raft_consensus.cc:2802] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:18.866246 29104 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250905 19:59:18.867406 28049 consensus_peers.cc:489] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 -> Peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299): Couldn't send request to peer 7e9c9f591f05464f9b097bf003d14f4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ed920d4c0144d62acf66f595bbf34fd. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:18.869509 29104 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:18.872784 28151 raft_consensus.cc:1273] T 5ed920d4c0144d62acf66f595bbf34fd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:18.874325 29160 consensus_queue.cc:1035] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:18.889793 28009 raft_consensus.cc:1273] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:18.892777 29160 consensus_queue.cc:1035] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:18.899194 29104 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:18.907088 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:18.907783 28079 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:18.919999 28079 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:59:18.934932 29104 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:18.936378 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:18.939251 27973 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:18.940467 29135 raft_consensus.cc:2953] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:18.942654 29185 raft_consensus.cc:2802] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:18.943502 29185 raft_consensus.cc:695] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:18.944521 29185 consensus_queue.cc:237] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:18.957804 28151 raft_consensus.cc:2953] T 5ed920d4c0144d62acf66f595bbf34fd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:18.962044 28002 raft_consensus.cc:1273] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250905 19:59:18.961810 28009 raft_consensus.cc:2953] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:18.967207 29160 consensus_queue.cc:1035] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:18.974524 28079 consensus_queue.cc:237] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:18.974241 27905 catalog_manager.cc:5582] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:19.008597 27905 catalog_manager.cc:5582] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 2 to 3, leader changed from 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 3 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
W20250905 19:59:19.027832 28049 consensus_peers.cc:489] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 -> Peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299): Couldn't send request to peer 7e9c9f591f05464f9b097bf003d14f4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f356a2522e646889c023d4b3d26f9cc. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:19.042024 28002 raft_consensus.cc:1273] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:19.044963 29079 consensus_queue.cc:1035] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:19.051759 28151 raft_consensus.cc:1273] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:19.057727 29079 consensus_queue.cc:1035] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:19.065699 28152 raft_consensus.cc:1273] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250905 19:59:19.074594 29079 consensus_queue.cc:1035] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20250905 19:59:19.073024 29193 raft_consensus.cc:2953] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:19.075794 28002 raft_consensus.cc:2953] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:19.085450 28079 raft_consensus.cc:1238] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:19.089407 29173 consensus_queue.cc:1046] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:19.090303 29174 raft_consensus.cc:3053] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:19.090744 29174 raft_consensus.cc:738] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:19.091600 29174 consensus_queue.cc:260] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:19.092927 29174 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:19.118347 28152 raft_consensus.cc:2953] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:19.127043 27908 catalog_manager.cc:5582] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:19.147157 28002 consensus_queue.cc:237] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
W20250905 19:59:19.167493 27974 consensus_peers.cc:489] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 -> Peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299): Couldn't send request to peer 7e9c9f591f05464f9b097bf003d14f4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b18b7149ff3439a8dde09149cf336b3. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:19.174710 28151 raft_consensus.cc:1273] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:19.176720 29185 consensus_queue.cc:1035] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:19.178583 28079 raft_consensus.cc:1273] T 1b18b7149ff3439a8dde09149cf336b3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:19.180028 29185 consensus_queue.cc:1035] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:19.208076 29185 raft_consensus.cc:2953] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:19.215703 28151 raft_consensus.cc:2953] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:19.226876 28080 consensus_queue.cc:237] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:19.230072 28082 raft_consensus.cc:2953] T 1b18b7149ff3439a8dde09149cf336b3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:19.238812 27905 catalog_manager.cc:5582] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:19.262224 28152 raft_consensus.cc:1273] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250905 19:59:19.262866 28049 consensus_peers.cc:489] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 -> Peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299): Couldn't send request to peer 7e9c9f591f05464f9b097bf003d14f4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 761b89d47d5543d89dd47af039cfe7de. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:19.263305 28002 raft_consensus.cc:1273] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:19.265695 29193 consensus_queue.cc:1035] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:19.267699 29193 consensus_queue.cc:1035] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:19.280553 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:59:19.295835 29193 raft_consensus.cc:2953] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:19.301496 28009 raft_consensus.cc:2953] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:19.323408 29202 ts_tablet_manager.cc:927] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a: Initiating tablet copy from peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:19.332827 28082 raft_consensus.cc:1273] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250905 19:59:19.329370 27908 catalog_manager.cc:5582] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:19.334860 29182 consensus_queue.cc:1035] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:19.364114 28151 raft_consensus.cc:2953] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:19.382732 28153 raft_consensus.cc:1273] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250905 19:59:19.383188 28153 pending_rounds.cc:85] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting all ops after (but not including) 1
I20250905 19:59:19.383458 28153 pending_rounds.cc:107] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250905 19:59:19.383678 28153 raft_consensus.cc:2887] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250905 19:59:19.392870 29104 consensus_queue.cc:1035] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:19.333648 29202 tablet_copy_client.cc:323] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Beginning tablet copy session from remote peer at address 127.26.58.130:34193
I20250905 19:59:19.426272 28095 tablet_copy_service.cc:140] P e4413ae5e0b64b52a6f360ac580239b9: Received BeginTabletCopySession request for tablet 5ed920d4c0144d62acf66f595bbf34fd from peer 7e9c9f591f05464f9b097bf003d14f4a ({username='slave'} at 127.0.0.1:50660)
I20250905 19:59:19.426996 28095 tablet_copy_service.cc:161] P e4413ae5e0b64b52a6f360ac580239b9: Beginning new tablet copy session on tablet 5ed920d4c0144d62acf66f595bbf34fd from peer 7e9c9f591f05464f9b097bf003d14f4a at {username='slave'} at 127.0.0.1:50660: session id = 7e9c9f591f05464f9b097bf003d14f4a-5ed920d4c0144d62acf66f595bbf34fd
I20250905 19:59:19.449394 28095 tablet_copy_source_session.cc:215] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:19.463464 29202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ed920d4c0144d62acf66f595bbf34fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:19.520181 28082 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5"
dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
mode: GRACEFUL
new_leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
from {username='slave'} at 127.0.0.1:51630
I20250905 19:59:19.520977 28082 raft_consensus.cc:604] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Received request to transfer leadership to TS 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:19.528086 29202 tablet_copy_client.cc:806] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:19.529191 29202 tablet_copy_client.cc:670] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:19.534266 29202 tablet_copy_client.cc:538] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 19:59:19.546254 29202 tablet_bootstrap.cc:492] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap starting.
I20250905 19:59:19.553449 28009 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5"
dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
mode: GRACEFUL
new_leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
from {username='slave'} at 127.0.0.1:47564
I20250905 19:59:19.554119 28009 raft_consensus.cc:604] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Received request to transfer leadership to TS 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:19.555869 29208 ts_tablet_manager.cc:927] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a: Initiating tablet copy from peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:19.560456 28082 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "6db33851e9914ca58560e8902701dca3"
dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
mode: GRACEFUL
new_leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:51630
I20250905 19:59:19.561118 28082 raft_consensus.cc:604] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Received request to transfer leadership to TS cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:19.564652 28009 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb"
dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
mode: GRACEFUL
new_leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
from {username='slave'} at 127.0.0.1:47564
I20250905 19:59:19.565301 28009 raft_consensus.cc:604] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Received request to transfer leadership to TS 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:19.569902 28002 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6"
dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
mode: GRACEFUL
new_leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
from {username='slave'} at 127.0.0.1:47564
I20250905 19:59:19.570652 28002 raft_consensus.cc:604] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 LEADER]: Received request to transfer leadership to TS 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:19.571837 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250905 19:59:19.573560 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:59:19.575860 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:59:19.579443 29202 tablet_bootstrap.cc:492] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:19.584187 29202 tablet_bootstrap.cc:492] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap complete.
I20250905 19:59:19.589041 29202 ts_tablet_manager.cc:1397] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a: Time spent bootstrapping tablet: real 0.043s user 0.019s sys 0.008s
I20250905 19:59:19.598384 29208 tablet_copy_client.cc:323] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Beginning tablet copy session from remote peer at address 127.26.58.130:34193
I20250905 19:59:19.599520 29202 raft_consensus.cc:357] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:19.600345 28095 tablet_copy_service.cc:140] P e4413ae5e0b64b52a6f360ac580239b9: Received BeginTabletCopySession request for tablet 2f356a2522e646889c023d4b3d26f9cc from peer 7e9c9f591f05464f9b097bf003d14f4a ({username='slave'} at 127.0.0.1:50660)
I20250905 19:59:19.601117 28095 tablet_copy_service.cc:161] P e4413ae5e0b64b52a6f360ac580239b9: Beginning new tablet copy session on tablet 2f356a2522e646889c023d4b3d26f9cc from peer 7e9c9f591f05464f9b097bf003d14f4a at {username='slave'} at 127.0.0.1:50660: session id = 7e9c9f591f05464f9b097bf003d14f4a-2f356a2522e646889c023d4b3d26f9cc
I20250905 19:59:19.600950 29202 raft_consensus.cc:738] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Initialized, Role: LEARNER
I20250905 19:59:19.602836 29202 consensus_queue.cc:260] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:19.616961 29202 ts_tablet_manager.cc:1428] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a: Time spent starting tablet: real 0.026s user 0.009s sys 0.005s
I20250905 19:59:19.627354 28094 tablet_copy_service.cc:342] P e4413ae5e0b64b52a6f360ac580239b9: Request end of tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-5ed920d4c0144d62acf66f595bbf34fd received from {username='slave'} at 127.0.0.1:50660
I20250905 19:59:19.627933 28094 tablet_copy_service.cc:434] P e4413ae5e0b64b52a6f360ac580239b9: ending tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-5ed920d4c0144d62acf66f595bbf34fd on tablet 5ed920d4c0144d62acf66f595bbf34fd with peer 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:19.641198 28095 tablet_copy_source_session.cc:215] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:19.659821 29208 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f356a2522e646889c023d4b3d26f9cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:19.680245 29202 ts_tablet_manager.cc:927] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a: Initiating tablet copy from peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:19.682555 29202 tablet_copy_client.cc:323] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Beginning tablet copy session from remote peer at address 127.26.58.130:34193
I20250905 19:59:19.684850 28095 tablet_copy_service.cc:140] P e4413ae5e0b64b52a6f360ac580239b9: Received BeginTabletCopySession request for tablet 761b89d47d5543d89dd47af039cfe7de from peer 7e9c9f591f05464f9b097bf003d14f4a ({username='slave'} at 127.0.0.1:50660)
I20250905 19:59:19.685482 28095 tablet_copy_service.cc:161] P e4413ae5e0b64b52a6f360ac580239b9: Beginning new tablet copy session on tablet 761b89d47d5543d89dd47af039cfe7de from peer 7e9c9f591f05464f9b097bf003d14f4a at {username='slave'} at 127.0.0.1:50660: session id = 7e9c9f591f05464f9b097bf003d14f4a-761b89d47d5543d89dd47af039cfe7de
I20250905 19:59:19.695746 29208 tablet_copy_client.cc:806] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:19.697026 29208 tablet_copy_client.cc:670] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:19.709312 28095 tablet_copy_source_session.cc:215] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:19.711855 29208 tablet_copy_client.cc:538] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 19:59:19.727970 29214 ts_tablet_manager.cc:927] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a: Initiating tablet copy from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421)
I20250905 19:59:19.729570 29202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 761b89d47d5543d89dd47af039cfe7de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:19.732652 29212 raft_consensus.cc:991] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9: : Instructing follower 26b2f874b8e54f08b01a9ed00d5cce05 to start an election
I20250905 19:59:19.733383 29194 raft_consensus.cc:1079] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Signalling peer 26b2f874b8e54f08b01a9ed00d5cce05 to start an election
I20250905 19:59:19.736549 28152 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
from {username='slave'} at 127.0.0.1:33756
I20250905 19:59:19.737401 28152 raft_consensus.cc:491] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:19.737891 28152 raft_consensus.cc:3058] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:19.749487 28152 raft_consensus.cc:513] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:19.755954 29208 tablet_bootstrap.cc:492] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap starting.
I20250905 19:59:19.763242 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:19.763931 28009 raft_consensus.cc:3058] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:19.766074 28152 leader_election.cc:290] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:19.766250 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:19.767067 28082 raft_consensus.cc:3053] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:19.767431 28082 raft_consensus.cc:738] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:19.770695 29202 tablet_copy_client.cc:806] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:19.770376 28082 consensus_queue.cc:260] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:19.771766 29202 tablet_copy_client.cc:670] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:19.772342 28082 raft_consensus.cc:3058] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:19.778478 28009 raft_consensus.cc:2466] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:19.779325 29202 tablet_copy_client.cc:538] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 19:59:19.784462 28125 leader_election.cc:304] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:19.785522 29174 raft_consensus.cc:2802] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:19.786003 29174 raft_consensus.cc:695] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:19.786885 29174 consensus_queue.cc:237] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:19.792161 29214 tablet_copy_client.cc:323] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Beginning tablet copy session from remote peer at address 127.26.58.129:38421
I20250905 19:59:19.798272 28019 tablet_copy_service.cc:140] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Received BeginTabletCopySession request for tablet 1b18b7149ff3439a8dde09149cf336b3 from peer 7e9c9f591f05464f9b097bf003d14f4a ({username='slave'} at 127.0.0.1:45844)
I20250905 19:59:19.798939 28019 tablet_copy_service.cc:161] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Beginning new tablet copy session on tablet 1b18b7149ff3439a8dde09149cf336b3 from peer 7e9c9f591f05464f9b097bf003d14f4a at {username='slave'} at 127.0.0.1:45844: session id = 7e9c9f591f05464f9b097bf003d14f4a-1b18b7149ff3439a8dde09149cf336b3
I20250905 19:59:19.799706 28082 raft_consensus.cc:2466] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:19.813889 29202 tablet_bootstrap.cc:492] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap starting.
I20250905 19:59:19.823891 28019 tablet_copy_source_session.cc:215] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:19.831856 29214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b18b7149ff3439a8dde09149cf336b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:19.847302 27905 catalog_manager.cc:5582] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 1 to 2, leader changed from e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 2 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20250905 19:59:19.863446 29133 raft_consensus.cc:991] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0: : Instructing follower 7e9c9f591f05464f9b097bf003d14f4a to start an election
I20250905 19:59:19.865029 29119 raft_consensus.cc:1079] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 LEADER]: Signalling peer 7e9c9f591f05464f9b097bf003d14f4a to start an election
I20250905 19:59:19.867198 28752 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6"
dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
from {username='slave'} at 127.0.0.1:40904
I20250905 19:59:19.867791 28752 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:19.868113 28752 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:19.877307 28752 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:19.880867 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:19.881560 28082 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:19.881965 28752 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:19.885829 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:19.886806 28003 raft_consensus.cc:3053] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:59:19.889146 28003 raft_consensus.cc:738] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:19.890333 28003 consensus_queue.cc:260] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:19.897528 28003 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:19.903313 29214 tablet_copy_client.cc:806] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:19.903990 29214 tablet_copy_client.cc:670] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:19.922405 29185 raft_consensus.cc:991] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0: : Instructing follower 26b2f874b8e54f08b01a9ed00d5cce05 to start an election
I20250905 19:59:19.923009 29185 raft_consensus.cc:1079] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Signalling peer 26b2f874b8e54f08b01a9ed00d5cce05 to start an election
I20250905 19:59:19.923476 28082 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 19:59:19.915684 28752 raft_consensus.cc:1215] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250905 19:59:19.911279 29214 tablet_copy_client.cc:538] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 19:59:19.932415 28152 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
from {username='slave'} at 127.0.0.1:33752
I20250905 19:59:19.934842 28152 raft_consensus.cc:491] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:19.935237 28152 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:19.940729 28715 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:19.941857 29211 raft_consensus.cc:2802] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:19.959568 28152 raft_consensus.cc:513] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:19.961587 29119 raft_consensus.cc:991] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0: : Instructing follower 7e9c9f591f05464f9b097bf003d14f4a to start an election
I20250905 19:59:19.962836 28003 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 19:59:19.962548 29211 raft_consensus.cc:695] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:19.963141 29119 raft_consensus.cc:1079] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Signalling peer 7e9c9f591f05464f9b097bf003d14f4a to start an election
I20250905 19:59:19.967787 29211 consensus_queue.cc:237] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:19.971267 28752 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5"
dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
from {username='slave'} at 127.0.0.1:40904
I20250905 19:59:19.972019 28752 raft_consensus.cc:491] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:19.973659 28152 leader_election.cc:290] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:19.975317 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:19.979538 28003 raft_consensus.cc:3053] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:19.979894 28003 raft_consensus.cc:738] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:19.979866 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:19.980818 28082 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:19.980952 28003 consensus_queue.cc:260] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:19.982232 28003 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:19.991848 29202 tablet_bootstrap.cc:492] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:19.993036 29202 tablet_bootstrap.cc:492] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap complete.
I20250905 19:59:19.997685 28082 raft_consensus.cc:2466] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:20.003484 28752 raft_consensus.cc:3058] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:20.012388 29202 ts_tablet_manager.cc:1397] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a: Time spent bootstrapping tablet: real 0.199s user 0.024s sys 0.011s
I20250905 19:59:20.014806 28122 leader_election.cc:304] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:20.017004 29174 raft_consensus.cc:2802] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:20.017596 29174 raft_consensus.cc:695] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:20.018570 29174 consensus_queue.cc:237] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:20.020452 28003 raft_consensus.cc:2466] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:20.029567 28752 raft_consensus.cc:513] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:20.031508 28752 leader_election.cc:290] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:20.033499 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:20.038801 28082 raft_consensus.cc:3058] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:20.045231 28082 raft_consensus.cc:2466] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 2.
I20250905 19:59:20.046674 28715 leader_election.cc:304] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:20.049556 29211 raft_consensus.cc:2802] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:20.050140 29211 raft_consensus.cc:695] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:20.051151 29211 consensus_queue.cc:237] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:20.052692 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:20.068310 28003 raft_consensus.cc:3053] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:20.070552 28003 raft_consensus.cc:738] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:20.072192 28003 consensus_queue.cc:260] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:20.073648 28003 raft_consensus.cc:3058] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:20.079654 27905 catalog_manager.cc:5582] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 3 to 4, leader changed from cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) to 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132). New cstate: current_term: 4 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 19:59:20.082818 29212 raft_consensus.cc:991] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9: : Instructing follower cd764bfeebac4cc9ad851cd4a6a7d3b0 to start an election
I20250905 19:59:20.083351 29212 raft_consensus.cc:1079] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Signalling peer cd764bfeebac4cc9ad851cd4a6a7d3b0 to start an election
I20250905 19:59:20.089058 28006 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "6db33851e9914ca58560e8902701dca3"
dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:54156
I20250905 19:59:20.090116 28006 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:20.090613 28006 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:20.092447 29208 tablet_bootstrap.cc:492] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:20.093420 29208 tablet_bootstrap.cc:492] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap complete.
I20250905 19:59:20.095918 29208 ts_tablet_manager.cc:1397] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a: Time spent bootstrapping tablet: real 0.340s user 0.024s sys 0.031s
I20250905 19:59:20.098246 29202 raft_consensus.cc:357] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:20.100409 29202 raft_consensus.cc:738] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Initialized, Role: LEARNER
I20250905 19:59:20.101186 29202 consensus_queue.cc:260] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:20.104754 28006 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:20.103205 29214 tablet_bootstrap.cc:492] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap starting.
I20250905 19:59:20.116400 28006 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:20.116287 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:20.119246 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:20.120007 28152 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:20.127812 28003 raft_consensus.cc:2466] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 2.
I20250905 19:59:20.128818 29202 ts_tablet_manager.cc:1428] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a: Time spent starting tablet: real 0.116s user 0.009s sys 0.000s
I20250905 19:59:20.135792 29208 raft_consensus.cc:357] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:20.139839 28095 tablet_copy_service.cc:342] P e4413ae5e0b64b52a6f360ac580239b9: Request end of tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-761b89d47d5543d89dd47af039cfe7de received from {username='slave'} at 127.0.0.1:50660
I20250905 19:59:20.140319 28095 tablet_copy_service.cc:434] P e4413ae5e0b64b52a6f360ac580239b9: ending tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-761b89d47d5543d89dd47af039cfe7de on tablet 761b89d47d5543d89dd47af039cfe7de with peer 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:20.146616 28152 raft_consensus.cc:2466] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 19:59:20.136734 29208 raft_consensus.cc:738] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Initialized, Role: LEARNER
I20250905 19:59:20.147665 28082 raft_consensus.cc:3053] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:59:20.148025 28082 raft_consensus.cc:738] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:20.148926 28082 consensus_queue.cc:260] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:20.150485 28082 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:20.153211 27975 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:20.155970 27905 catalog_manager.cc:5582] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 1 to 2, leader changed from cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 2 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20250905 19:59:20.158579 28082 raft_consensus.cc:2466] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 19:59:20.156759 29133 raft_consensus.cc:2802] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:20.159499 29133 raft_consensus.cc:695] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:20.150835 29208 consensus_queue.cc:260] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:20.160450 29133 consensus_queue.cc:237] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:20.171377 29208 ts_tablet_manager.cc:1428] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a: Time spent starting tablet: real 0.075s user 0.006s sys 0.002s
I20250905 19:59:20.176115 28095 tablet_copy_service.cc:342] P e4413ae5e0b64b52a6f360ac580239b9: Request end of tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-2f356a2522e646889c023d4b3d26f9cc received from {username='slave'} at 127.0.0.1:50660
I20250905 19:59:20.176636 28095 tablet_copy_service.cc:434] P e4413ae5e0b64b52a6f360ac580239b9: ending tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-2f356a2522e646889c023d4b3d26f9cc on tablet 2f356a2522e646889c023d4b3d26f9cc with peer 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:20.192094 27906 catalog_manager.cc:5582] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 3 to 4, leader changed from e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 4 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:20.193475 27908 catalog_manager.cc:5582] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 1 to 2, leader changed from cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) to 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132). New cstate: current_term: 2 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 19:59:20.227604 28082 raft_consensus.cc:1273] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:20.229326 29173 consensus_queue.cc:1035] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:20.252750 29214 tablet_bootstrap.cc:492] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:20.254143 29214 tablet_bootstrap.cc:492] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap complete.
I20250905 19:59:20.255802 29214 ts_tablet_manager.cc:1397] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent bootstrapping tablet: real 0.159s user 0.034s sys 0.005s
I20250905 19:59:20.376842 29214 raft_consensus.cc:357] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:20.380057 29214 raft_consensus.cc:738] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Initialized, Role: LEARNER
I20250905 19:59:20.380913 29214 consensus_queue.cc:260] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:20.388020 29214 ts_tablet_manager.cc:1428] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent starting tablet: real 0.132s user 0.019s sys 0.022s
I20250905 19:59:20.402036 28080 raft_consensus.cc:1273] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250905 19:59:20.404793 29211 consensus_queue.cc:1035] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250905 19:59:20.411412 28749 raft_consensus.cc:1215] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250905 19:59:20.416417 28019 tablet_copy_service.cc:342] P cd764bfeebac4cc9ad851cd4a6a7d3b0: Request end of tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-1b18b7149ff3439a8dde09149cf336b3 received from {username='slave'} at 127.0.0.1:45844
I20250905 19:59:20.417125 28019 tablet_copy_service.cc:434] P cd764bfeebac4cc9ad851cd4a6a7d3b0: ending tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-1b18b7149ff3439a8dde09149cf336b3 on tablet 1b18b7149ff3439a8dde09149cf336b3 with peer 7e9c9f591f05464f9b097bf003d14f4a
W20250905 19:59:20.416376 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 43897us
I20250905 19:59:20.451203 28002 raft_consensus.cc:1273] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250905 19:59:20.452937 29211 consensus_queue.cc:1035] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250905 19:59:20.472108 28006 raft_consensus.cc:1273] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:20.497885 29231 consensus_queue.cc:1035] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:20.501905 28002 raft_consensus.cc:1273] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:20.518399 29232 consensus_queue.cc:1035] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:20.533167 28006 raft_consensus.cc:1273] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250905 19:59:20.540479 29193 raft_consensus.cc:1062] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9: attempting to promote NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a to VOTER
I20250905 19:59:20.542459 29238 consensus_queue.cc:1035] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250905 19:59:20.543821 29193 consensus_queue.cc:237] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:20.621379 28081 raft_consensus.cc:1273] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:20.625001 29174 consensus_queue.cc:1035] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20250905 19:59:20.661091 28151 raft_consensus.cc:1273] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250905 19:59:20.669768 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20250905 19:59:20.703469 29243 log.cc:927] Time spent T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.079s user 0.001s sys 0.000s
I20250905 19:59:20.716174 28081 raft_consensus.cc:1273] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250905 19:59:20.738188 29133 consensus_queue.cc:1035] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:20.741078 29238 consensus_queue.cc:1035] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20250905 19:59:20.763778 28752 raft_consensus.cc:1215] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250905 19:59:20.781935 28080 raft_consensus.cc:1273] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250905 19:59:20.783576 29119 consensus_queue.cc:1035] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:20.797530 28084 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d"
dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
mode: GRACEFUL
new_leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
from {username='slave'} at 127.0.0.1:51630
I20250905 19:59:20.798193 28084 raft_consensus.cc:604] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Received request to transfer leadership to TS 26b2f874b8e54f08b01a9ed00d5cce05
W20250905 19:59:20.840626 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 89951.6us
I20250905 19:59:20.856786 28752 raft_consensus.cc:1273] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:20.858927 28153 raft_consensus.cc:1273] T 5ed920d4c0144d62acf66f595bbf34fd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:20.860358 29160 consensus_queue.cc:1035] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:20.862720 29079 consensus_queue.cc:1035] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:20.890944 28749 raft_consensus.cc:1215] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250905 19:59:20.912123 29079 raft_consensus.cc:2953] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:20.917200 28749 raft_consensus.cc:2953] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:20.924557 29160 raft_consensus.cc:1062] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9: attempting to promote NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a to VOTER
I20250905 19:59:20.935987 29160 consensus_queue.cc:237] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:20.950260 28159 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:36164
I20250905 19:59:20.950910 28159 raft_consensus.cc:604] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Received request to transfer leadership to TS cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:20.957242 28006 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d"
dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
mode: GRACEFUL
new_leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
from {username='slave'} at 127.0.0.1:47564
I20250905 19:59:20.957934 28006 raft_consensus.cc:604] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Received request to transfer leadership to TS 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:20.965268 28159 raft_consensus.cc:2953] T 5ed920d4c0144d62acf66f595bbf34fd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:20.970808 28082 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "e707b0f561934674947fb463ce626a07"
dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
mode: GRACEFUL
new_leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
from {username='slave'} at 127.0.0.1:51630
I20250905 19:59:20.971774 28082 raft_consensus.cc:604] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Received request to transfer leadership to TS 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:20.981967 27906 catalog_manager.cc:5582] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:20.983480 28151 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "fd2b679c53994263846f51ceec97a150"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
from {username='slave'} at 127.0.0.1:36164
I20250905 19:59:20.988374 28151 raft_consensus.cc:604] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Received request to transfer leadership to TS 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:20.991014 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250905 19:59:20.992786 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:59:20.994594 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:59:21.030067 29079 raft_consensus.cc:991] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9: : Instructing follower 26b2f874b8e54f08b01a9ed00d5cce05 to start an election
I20250905 19:59:21.030673 29079 raft_consensus.cc:1079] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Signalling peer 26b2f874b8e54f08b01a9ed00d5cce05 to start an election
I20250905 19:59:21.034998 28151 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
from {username='slave'} at 127.0.0.1:33756
I20250905 19:59:21.035620 28151 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:21.035961 28151 raft_consensus.cc:3058] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:21.051174 28151 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:21.056447 28006 raft_consensus.cc:1273] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:21.060225 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:21.061604 28006 raft_consensus.cc:3058] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:21.064919 28151 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:21.065573 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:21.066267 28082 raft_consensus.cc:3053] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:21.066681 28082 raft_consensus.cc:738] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:21.068147 28082 consensus_queue.cc:260] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:21.069306 28082 raft_consensus.cc:3058] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:21.072801 28749 raft_consensus.cc:1273] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:21.074187 29212 consensus_queue.cc:1035] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:21.077759 28082 raft_consensus.cc:2466] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:21.079126 28122 leader_election.cc:304] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:21.080082 29089 raft_consensus.cc:2802] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:21.080529 29089 raft_consensus.cc:695] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:21.081276 29089 consensus_queue.cc:237] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:21.084815 28151 raft_consensus.cc:1273] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:21.085350 28006 raft_consensus.cc:2466] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:21.104203 29231 raft_consensus.cc:991] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05: : Instructing follower 7e9c9f591f05464f9b097bf003d14f4a to start an election
I20250905 19:59:21.104761 29231 raft_consensus.cc:1079] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Signalling peer 7e9c9f591f05464f9b097bf003d14f4a to start an election
I20250905 19:59:21.086557 29160 consensus_queue.cc:1035] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:21.090619 28009 raft_consensus.cc:1273] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:21.101433 29079 raft_consensus.cc:1062] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9: attempting to promote NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a to VOTER
I20250905 19:59:21.108291 28749 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "fd2b679c53994263846f51ceec97a150"
dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
from {username='slave'} at 127.0.0.1:40926
I20250905 19:59:21.110358 28749 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:21.110872 28749 raft_consensus.cc:3058] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:21.116262 28749 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:21.121635 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:21.122512 28006 raft_consensus.cc:3058] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:21.124987 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:21.125771 28151 raft_consensus.cc:3053] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:59:21.126300 28151 raft_consensus.cc:738] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:21.127681 28151 consensus_queue.cc:260] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:21.126816 28749 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:21.136898 28151 raft_consensus.cc:3058] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:21.130213 28006 raft_consensus.cc:2466] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 19:59:21.141363 28718 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:21.142505 29236 raft_consensus.cc:2802] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:21.143016 29236 raft_consensus.cc:695] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:21.143857 29236 consensus_queue.cc:237] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:21.148651 28151 raft_consensus.cc:2466] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
W20250905 19:59:21.159209 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 56335.4us
I20250905 19:59:21.202194 29089 raft_consensus.cc:991] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05: : Instructing follower cd764bfeebac4cc9ad851cd4a6a7d3b0 to start an election
I20250905 19:59:21.213363 29089 raft_consensus.cc:1079] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Signalling peer cd764bfeebac4cc9ad851cd4a6a7d3b0 to start an election
I20250905 19:59:21.217192 29271 consensus_queue.cc:1035] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:21.216485 29193 raft_consensus.cc:2953] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:21.226346 28006 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1"
dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:54164
I20250905 19:59:21.227061 28006 raft_consensus.cc:491] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:21.227404 28006 raft_consensus.cc:3058] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:21.237955 28006 raft_consensus.cc:513] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:21.243496 28006 leader_election.cc:290] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:21.247030 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:21.249282 28153 raft_consensus.cc:3053] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:21.249619 28153 raft_consensus.cc:738] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:21.221864 29079 consensus_queue.cc:237] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:21.250617 28153 consensus_queue.cc:260] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:21.259866 28153 raft_consensus.cc:3058] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:21.265267 29254 raft_consensus.cc:1062] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: attempting to promote NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a to VOTER
I20250905 19:59:21.269510 29254 consensus_queue.cc:237] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:21.279036 28153 raft_consensus.cc:2466] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:21.282279 27975 leader_election.cc:304] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:21.283936 29182 raft_consensus.cc:2802] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:21.284695 29182 raft_consensus.cc:695] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:21.286041 29182 consensus_queue.cc:237] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:21.296756 27906 catalog_manager.cc:5582] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 3 to 4, leader changed from 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) to 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132). New cstate: current_term: 4 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 19:59:21.341863 27905 catalog_manager.cc:5582] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 1 to 2, leader changed from e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 2 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
W20250905 19:59:21.391134 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 252969us
I20250905 19:59:21.403337 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:21.403964 28082 raft_consensus.cc:3058] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:21.421169 28752 raft_consensus.cc:2953] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:21.457702 28082 raft_consensus.cc:2466] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
W20250905 19:59:21.456694 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 270960us
W20250905 19:59:21.470638 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 43972us
W20250905 19:59:21.479682 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 46808.1us
I20250905 19:59:21.488131 29185 raft_consensus.cc:991] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0: : Instructing follower 7e9c9f591f05464f9b097bf003d14f4a to start an election
I20250905 19:59:21.488677 29185 raft_consensus.cc:1079] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Signalling peer 7e9c9f591f05464f9b097bf003d14f4a to start an election
I20250905 19:59:21.505803 27905 catalog_manager.cc:5582] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:21.512107 28752 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d"
dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
from {username='slave'} at 127.0.0.1:40904
I20250905 19:59:21.512743 28752 raft_consensus.cc:491] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:21.513164 28752 raft_consensus.cc:3058] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Advancing to term 2
W20250905 19:59:21.514966 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 43646.8us
I20250905 19:59:21.519507 28749 raft_consensus.cc:1273] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:21.523366 28751 raft_consensus.cc:1273] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20250905 19:59:21.531108 29278 log.cc:927] Time spent T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
W20250905 19:59:21.532703 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 116591us
I20250905 19:59:21.539958 29265 consensus_queue.cc:1035] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:21.544672 28752 raft_consensus.cc:513] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:21.547289 28151 raft_consensus.cc:2953] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:21.547274 28752 leader_election.cc:290] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:21.551913 29160 consensus_queue.cc:1035] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:21.568126 28082 raft_consensus.cc:1273] T 1b18b7149ff3439a8dde09149cf336b3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20250905 19:59:21.611006 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 48871.4us
I20250905 19:59:21.558475 29185 consensus_queue.cc:1035] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20250905 19:59:21.633412 28080 raft_consensus.cc:1273] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:21.635646 28151 raft_consensus.cc:1273] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250905 19:59:21.637415 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:21.638018 28151 raft_consensus.cc:3058] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:21.652104 28004 raft_consensus.cc:1273] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:21.657562 28156 raft_consensus.cc:1273] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:21.663379 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:21.664067 28004 raft_consensus.cc:3053] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:21.664357 28004 raft_consensus.cc:738] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:21.665071 28004 consensus_queue.cc:260] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:21.668618 28004 raft_consensus.cc:3058] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:21.670867 28151 raft_consensus.cc:2466] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 2.
I20250905 19:59:21.672135 28717 leader_election.cc:304] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 19:59:21.641482 29236 consensus_queue.cc:1035] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20250905 19:59:21.673691 29282 consensus_queue.cc:1035] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20250905 19:59:21.672706 28006 raft_consensus.cc:2953] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
W20250905 19:59:21.698611 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 60080.7us
I20250905 19:59:21.703918 29079 raft_consensus.cc:991] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9: : Instructing follower 26b2f874b8e54f08b01a9ed00d5cce05 to start an election
W20250905 19:59:21.740005 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 65239.8us
I20250905 19:59:21.724557 29288 consensus_queue.cc:1035] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20250905 19:59:21.713896 29265 raft_consensus.cc:1079] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Signalling peer 26b2f874b8e54f08b01a9ed00d5cce05 to start an election
W20250905 19:59:21.813894 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 92060.7us
W20250905 19:59:21.826474 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 91006.7us
I20250905 19:59:21.861506 28004 raft_consensus.cc:2466] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 2.
I20250905 19:59:21.869385 28009 raft_consensus.cc:1273] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:21.874730 29272 consensus_queue.cc:1035] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20250905 19:59:21.948902 27906 catalog_manager.cc:5582] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 1 to 2, leader changed from 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 2 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
W20250905 19:59:21.962567 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 108986us
I20250905 19:59:21.964586 28156 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "e707b0f561934674947fb463ce626a07"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
from {username='slave'} at 127.0.0.1:33756
I20250905 19:59:21.965214 28156 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:21.965560 28156 raft_consensus.cc:3058] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
W20250905 19:59:22.015846 29289 log.cc:927] Time spent T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
W20250905 19:59:22.020364 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 348265us
W20250905 19:59:22.035090 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 396917us
I20250905 19:59:22.035799 29222 raft_consensus.cc:2802] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:22.052294 28007 raft_consensus.cc:2953] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:22.090709 29222 raft_consensus.cc:695] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 2 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:22.091848 29222 consensus_queue.cc:237] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:22.095530 29294 log.cc:927] Time spent T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.063s user 0.000s sys 0.002s
W20250905 19:59:22.097597 29296 log.cc:927] Time spent T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20250905 19:59:22.104203 28156 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:22.105306 28153 raft_consensus.cc:1238] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
W20250905 19:59:22.108994 29290 log.cc:927] Time spent T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.245s user 0.005s sys 0.000s
I20250905 19:59:22.109617 29160 consensus_queue.cc:1046] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:22.110894 29160 raft_consensus.cc:3053] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:22.111307 29160 raft_consensus.cc:738] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:22.112242 29160 consensus_queue.cc:260] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:22.113873 29160 raft_consensus.cc:3058] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:22.123535 28156 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:22.139298 27905 catalog_manager.cc:5582] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 1 to 2, leader changed from cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) to 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132). New cstate: current_term: 2 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250905 19:59:22.152988 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 124655us
W20250905 19:59:22.133149 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 87809.3us
I20250905 19:59:22.157881 28153 raft_consensus.cc:1273] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:22.163579 29173 consensus_queue.cc:1035] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:22.166919 28153 raft_consensus.cc:1273] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20250905 19:59:22.171362 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 245993us
I20250905 19:59:22.188321 29280 raft_consensus.cc:2953] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:22.224359 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:22.228379 28080 raft_consensus.cc:1273] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250905 19:59:22.237119 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 45303.2us
I20250905 19:59:22.240002 28156 raft_consensus.cc:2953] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:22.257803 28749 raft_consensus.cc:2953] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:22.272819 29308 consensus_queue.cc:1035] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20250905 19:59:22.275493 29255 consensus_queue.cc:1035] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20250905 19:59:22.285909 29089 consensus_queue.cc:1035] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:22.291852 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:22.292579 28001 raft_consensus.cc:3058] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:22.297809 28003 raft_consensus.cc:2953] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:22.307948 28085 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:22.313915 28122 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:22.317698 29089 raft_consensus.cc:2802] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:22.318225 29089 raft_consensus.cc:695] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:22.319746 29089 consensus_queue.cc:237] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:22.363006 28001 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:22.346977 29308 consensus_queue.cc:1035] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20250905 19:59:22.376937 28001 raft_consensus.cc:1273] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250905 19:59:22.392486 29291 consensus_queue.cc:1035] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20250905 19:59:22.466333 29302 raft_consensus.cc:2953] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:22.474165 28154 raft_consensus.cc:2953] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:22.519901 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:59:22.498273 28083 raft_consensus.cc:2953] T 1b18b7149ff3439a8dde09149cf336b3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
W20250905 19:59:22.631178 29325 log.cc:927] Time spent T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20250905 19:59:22.632792 28154 raft_consensus.cc:1273] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20250905 19:59:22.655102 29323 log.cc:927] Time spent T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.071s user 0.001s sys 0.000s
I20250905 19:59:22.639711 29292 consensus_queue.cc:1035] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
W20250905 19:59:22.867655 29324 log.cc:927] Time spent T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.294s user 0.002s sys 0.000s
I20250905 19:59:23.218888 28004 raft_consensus.cc:1273] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20250905 19:59:23.252797 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 189058us
I20250905 19:59:23.272161 28082 raft_consensus.cc:1273] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:23.277119 29279 consensus_queue.cc:1035] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:23.297782 28008 raft_consensus.cc:1273] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:23.302584 27905 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:60818 (request call id 121) took 1167 ms. Trace:
I20250905 19:59:23.303020 27905 rpcz_store.cc:276] 0905 19:59:22.134802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:22.134959 (+ 157us) service_pool.cc:225] Handling call
0905 19:59:23.302485 (+1167526us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:59:22.147017 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:59:22.147593 (+ 576us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:59:22.147630 (+ 37us) write_op.cc:435] Acquired schema lock
0905 19:59:22.147649 (+ 19us) tablet.cc:660] Decoding operations
0905 19:59:22.148010 (+ 361us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:59:22.148067 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0905 19:59:22.148100 (+ 33us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:59:22.148297 (+ 197us) tablet.cc:699] Row locks acquired
0905 19:59:22.148314 (+ 17us) write_op.cc:260] PREPARE: finished
0905 19:59:22.148469 (+ 155us) write_op.cc:270] Start()
0905 19:59:22.148609 (+ 140us) write_op.cc:276] Timestamp: P: 1757102362148418 usec, L: 0
0905 19:59:22.148629 (+ 20us) op_driver.cc:348] REPLICATION: starting
0905 19:59:22.149337 (+ 708us) log.cc:844] Serialized 536 byte log entry
0905 19:59:22.428759 (+279422us) op_driver.cc:464] REPLICATION: finished
0905 19:59:22.565950 (+137191us) write_op.cc:301] APPLY: starting
0905 19:59:22.568924 (+ 2974us) tablet.cc:1365] starting BulkCheckPresence
0905 19:59:22.573944 (+ 5020us) tablet.cc:1368] finished BulkCheckPresence
0905 19:59:22.606637 (+ 32693us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:59:22.646228 (+ 39591us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:59:22.659727 (+ 13499us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:59:22.686580 (+ 26853us) write_op.cc:312] APPLY: finished
0905 19:59:22.686869 (+ 289us) log.cc:844] Serialized 23 byte log entry
0905 19:59:22.687118 (+ 249us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:59:22.687203 (+ 85us) write_op.cc:454] Released schema lock
0905 19:59:22.687239 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0905 19:59:22.687635 (+ 396us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:59:22.687667 (+ 32us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":60899,"apply.run_cpu_time_us":2698,"apply.run_wall_time_us":164941,"num_ops":1,"prepare.queue_time_us":132,"prepare.run_cpu_time_us":2574,"prepare.run_wall_time_us":2574,"replication_time_us":279976,"thread_start_us":33195,"threads_started":1}]]}
I20250905 19:59:23.308249 28083 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3"
dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
mode: GRACEFUL
new_leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:51630
I20250905 19:59:23.309055 28083 raft_consensus.cc:604] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Received request to transfer leadership to TS cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:23.310377 27906 catalog_manager.cc:5582] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 19:59:23.314985 28748 raft_consensus.cc:2953] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:23.361667 29292 consensus_queue.cc:1035] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250905 19:59:23.384207 28008 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "6db33851e9914ca58560e8902701dca3"
dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
mode: GRACEFUL
new_leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
from {username='slave'} at 127.0.0.1:47564
I20250905 19:59:23.385154 28008 raft_consensus.cc:604] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Received request to transfer leadership to TS 26b2f874b8e54f08b01a9ed00d5cce05
W20250905 19:59:23.469414 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 103810us
I20250905 19:59:23.471606 29231 consensus_queue.cc:1035] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:23.491704 29255 raft_consensus.cc:991] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: : Instructing follower 26b2f874b8e54f08b01a9ed00d5cce05 to start an election
I20250905 19:59:23.492313 29255 raft_consensus.cc:1079] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Signalling peer 26b2f874b8e54f08b01a9ed00d5cce05 to start an election
W20250905 19:59:23.504618 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 49673.1us
W20250905 19:59:23.521391 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 161086us
I20250905 19:59:23.523775 28155 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
from {username='slave'} at 127.0.0.1:36164
I20250905 19:59:23.524623 28155 raft_consensus.cc:604] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Received request to transfer leadership to TS 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:23.532440 28008 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d"
dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
mode: GRACEFUL
new_leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
from {username='slave'} at 127.0.0.1:47564
I20250905 19:59:23.533334 28008 raft_consensus.cc:604] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Received request to transfer leadership to TS 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:23.564080 29231 raft_consensus.cc:991] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05: : Instructing follower 7e9c9f591f05464f9b097bf003d14f4a to start an election
I20250905 19:59:23.564580 29231 raft_consensus.cc:1079] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Signalling peer 7e9c9f591f05464f9b097bf003d14f4a to start an election
I20250905 19:59:23.580721 28153 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "6db33851e9914ca58560e8902701dca3"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
from {username='slave'} at 127.0.0.1:33752
I20250905 19:59:23.581247 28082 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801"
dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
mode: GRACEFUL
new_leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:51630
I20250905 19:59:23.581310 28153 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:23.582167 28082 raft_consensus.cc:604] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Received request to transfer leadership to TS cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:23.582232 28153 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:23.583091 28749 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c"
dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
from {username='slave'} at 127.0.0.1:40926
I20250905 19:59:23.584324 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250905 19:59:23.586057 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:59:23.583863 28749 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:23.587226 28749 raft_consensus.cc:3058] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:23.596398 28153 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:23.615247 29307 raft_consensus.cc:991] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9: : Instructing follower cd764bfeebac4cc9ad851cd4a6a7d3b0 to start an election
I20250905 19:59:23.621690 28150 raft_consensus.cc:1238] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 4. Current term is 5. Ops: []
I20250905 19:59:23.623199 28153 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:23.624737 29307 raft_consensus.cc:1079] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Signalling peer cd764bfeebac4cc9ad851cd4a6a7d3b0 to start an election
W20250905 19:59:23.625610 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 69212.7us
I20250905 19:59:23.632124 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:23.634238 28082 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:23.594254 28749 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:23.647428 28749 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:23.646283 27908 catalog_manager.cc:5582] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 19:59:23.649168 28083 consensus_queue.cc:237] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:23.650079 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:23.650852 28085 raft_consensus.cc:3058] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:23.659083 27906 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34446 (request call id 213) took 1050 ms. Trace:
I20250905 19:59:23.660092 28003 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3"
dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:54156
I20250905 19:59:23.660671 28003 raft_consensus.cc:491] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:23.661029 28003 raft_consensus.cc:3058] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:23.659633 27906 rpcz_store.cc:276] 0905 19:59:22.609399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:22.609611 (+ 212us) service_pool.cc:225] Handling call
0905 19:59:23.659019 (+1049408us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:59:23.491170 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:59:23.491735 (+ 565us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:59:23.491933 (+ 198us) write_op.cc:435] Acquired schema lock
0905 19:59:23.491946 (+ 13us) tablet.cc:660] Decoding operations
0905 19:59:23.492285 (+ 339us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:59:23.492369 (+ 84us) write_op.cc:641] Partition lock acquired for write op
0905 19:59:23.492399 (+ 30us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:59:23.492597 (+ 198us) tablet.cc:699] Row locks acquired
0905 19:59:23.492615 (+ 18us) write_op.cc:260] PREPARE: finished
0905 19:59:23.492773 (+ 158us) write_op.cc:270] Start()
0905 19:59:23.492879 (+ 106us) write_op.cc:276] Timestamp: P: 1757102363492715 usec, L: 0
0905 19:59:23.492895 (+ 16us) op_driver.cc:348] REPLICATION: starting
0905 19:59:23.493619 (+ 724us) log.cc:844] Serialized 599 byte log entry
0905 19:59:23.566194 (+ 72575us) op_driver.cc:464] REPLICATION: finished
0905 19:59:23.633209 (+ 67015us) write_op.cc:301] APPLY: starting
0905 19:59:23.633293 (+ 84us) tablet.cc:1365] starting BulkCheckPresence
0905 19:59:23.633382 (+ 89us) tablet.cc:1368] finished BulkCheckPresence
0905 19:59:23.633505 (+ 123us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:59:23.633701 (+ 196us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:59:23.633851 (+ 150us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:59:23.633901 (+ 50us) write_op.cc:312] APPLY: finished
0905 19:59:23.634092 (+ 191us) log.cc:844] Serialized 23 byte log entry
0905 19:59:23.634260 (+ 168us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:59:23.634368 (+ 108us) write_op.cc:454] Released schema lock
0905 19:59:23.634406 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0905 19:59:23.634769 (+ 363us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:59:23.634798 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":178913,"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":56943,"apply.run_cpu_time_us":1547,"apply.run_wall_time_us":1551,"num_ops":1,"prepare.queue_time_us":163658,"prepare.run_cpu_time_us":2701,"prepare.run_wall_time_us":58607,"replication_time_us":73152,"thread_start_us":231618,"threads_started":2}]]}
I20250905 19:59:23.665308 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:59:23.684136 29337 consensus_queue.cc:1046] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:59:23.685571 29302 raft_consensus.cc:3053] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Stepping down as leader of term 4
I20250905 19:59:23.686018 29302 raft_consensus.cc:738] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:23.686964 29302 consensus_queue.cc:260] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:23.688967 29302 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:23.668347 28085 raft_consensus.cc:2466] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 2.
I20250905 19:59:23.672163 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:23.672068 28003 raft_consensus.cc:513] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:23.677950 29341 raft_consensus.cc:991] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9: : Instructing follower cd764bfeebac4cc9ad851cd4a6a7d3b0 to start an election
I20250905 19:59:23.696885 29341 raft_consensus.cc:1079] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Signalling peer cd764bfeebac4cc9ad851cd4a6a7d3b0 to start an election
I20250905 19:59:23.698110 28715 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:23.699018 28003 leader_election.cc:290] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:23.699050 29292 raft_consensus.cc:2802] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:23.699746 29292 raft_consensus.cc:695] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 2 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:23.700626 29292 consensus_queue.cc:237] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:23.678916 27907 catalog_manager.cc:5582] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 1 to 2, leader changed from e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 2 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20250905 19:59:23.714627 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:23.715221 28153 raft_consensus.cc:3053] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:23.715538 28153 raft_consensus.cc:738] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:23.716356 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:23.716225 28153 consensus_queue.cc:260] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:23.717649 28153 raft_consensus.cc:3058] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:23.720100 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:23.720695 28155 raft_consensus.cc:3058] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:23.720965 28003 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801"
dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:54156
I20250905 19:59:23.721514 28003 raft_consensus.cc:491] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:23.721920 28003 raft_consensus.cc:3058] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:23.724035 28082 raft_consensus.cc:2466] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 19:59:23.732162 28153 raft_consensus.cc:2466] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 2.
I20250905 19:59:23.734474 28085 raft_consensus.cc:3053] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:23.736517 28749 raft_consensus.cc:1273] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:23.739428 28002 consensus_queue.cc:237] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:23.741783 28150 raft_consensus.cc:1273] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:23.749202 29280 consensus_queue.cc:1035] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 19:59:23.753207 28748 raft_consensus.cc:1273] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:23.754813 29255 consensus_queue.cc:1035] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:59:23.725682 28122 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:23.757428 29174 raft_consensus.cc:2802] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election won for term 5
I20250905 19:59:23.758582 28085 raft_consensus.cc:738] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:23.760022 28155 raft_consensus.cc:2466] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:23.761157 28085 consensus_queue.cc:260] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:23.759482 29174 raft_consensus.cc:695] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:23.766552 29308 raft_consensus.cc:2953] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:23.772076 28003 raft_consensus.cc:513] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:23.773231 28748 raft_consensus.cc:2953] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:23.775463 28003 leader_election.cc:290] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:23.780292 29174 consensus_queue.cc:237] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:23.768981 28085 raft_consensus.cc:3058] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:23.799520 28083 consensus_queue.cc:237] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:23.799963 28080 consensus_queue.cc:237] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:23.787496 28004 raft_consensus.cc:2466] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 19:59:23.790313 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:23.804356 28082 raft_consensus.cc:3053] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:23.804723 28082 raft_consensus.cc:738] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:23.805439 28082 consensus_queue.cc:260] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:23.806545 28082 raft_consensus.cc:3058] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:23.791018 29341 consensus_queue.cc:1035] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250905 19:59:23.793965 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 52231.1us
I20250905 19:59:23.795766 27904 catalog_manager.cc:5582] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 1 to 2, leader changed from 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) to 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132). New cstate: current_term: 2 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 19:59:23.829257 28082 raft_consensus.cc:2466] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:23.830963 28749 raft_consensus.cc:2953] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:23.825472 29331 raft_consensus.cc:2953] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:23.839092 27973 leader_election.cc:304] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:23.843114 28747 raft_consensus.cc:1273] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:23.861841 27905 catalog_manager.cc:5582] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 4 to 5, leader changed from cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 5 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
W20250905 19:59:23.875200 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 52653us
I20250905 19:59:23.875941 27975 leader_election.cc:304] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:23.877875 27895 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 1b18b7149ff3439a8dde09149cf336b3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 19:59:23.879120 29338 raft_consensus.cc:2802] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:23.879650 29338 raft_consensus.cc:695] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:23.880467 29338 consensus_queue.cc:237] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:23.892534 28004 raft_consensus.cc:1273] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:23.858558 29307 consensus_queue.cc:1035] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
W20250905 19:59:23.906700 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 67733.4us
I20250905 19:59:23.908666 29308 raft_consensus.cc:2802] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:23.909174 29308 raft_consensus.cc:695] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:23.889752 27905 catalog_manager.cc:5582] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New cstate: current_term: 1 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:23.918881 28150 raft_consensus.cc:1273] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:23.919963 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:23.920557 28155 raft_consensus.cc:3058] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:23.941393 28155 raft_consensus.cc:2466] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:23.945475 28748 raft_consensus.cc:1273] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:23.910043 29308 consensus_queue.cc:237] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:23.933450 29336 consensus_queue.cc:1035] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
I20250905 19:59:23.972417 29255 raft_consensus.cc:991] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0: : Instructing follower 26b2f874b8e54f08b01a9ed00d5cce05 to start an election
I20250905 19:59:23.973135 29255 raft_consensus.cc:1079] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Signalling peer 26b2f874b8e54f08b01a9ed00d5cce05 to start an election
I20250905 19:59:23.978554 28157 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
from {username='slave'} at 127.0.0.1:33752
I20250905 19:59:23.979315 28157 raft_consensus.cc:491] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:23.979799 28157 raft_consensus.cc:3058] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
W20250905 19:59:23.979375 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 89564.1us
W20250905 19:59:23.987257 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 99600.1us
I20250905 19:59:23.989735 28004 raft_consensus.cc:1273] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:24.019753 29280 consensus_queue.cc:1035] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:59:24.023061 27892 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 2f356a2522e646889c023d4b3d26f9cc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 19:59:24.023324 28153 raft_consensus.cc:2953] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:24.062935 28155 raft_consensus.cc:2953] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
W20250905 19:59:24.071063 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 47818.3us
I20250905 19:59:24.034144 29190 consensus_queue.cc:1035] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
I20250905 19:59:24.073450 28085 raft_consensus.cc:2466] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:24.077576 29280 consensus_queue.cc:1035] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20250905 19:59:24.090010 28031 debug-util.cc:398] Leaking SignalData structure 0x7b08001bc6e0 after lost signal to thread 26861
W20250905 19:59:24.091163 28031 debug-util.cc:398] Leaking SignalData structure 0x7b08000d4da0 after lost signal to thread 27942
W20250905 19:59:24.092808 28031 debug-util.cc:398] Leaking SignalData structure 0x7b08002bdf20 after lost signal to thread 28034
I20250905 19:59:24.104642 28157 raft_consensus.cc:513] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:24.122048 28157 leader_election.cc:290] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 19:59:24.127882 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 99639.1us
I20250905 19:59:24.149233 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:24.149868 28082 raft_consensus.cc:3058] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:24.162020 28082 raft_consensus.cc:2466] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:24.238986 28085 raft_consensus.cc:1273] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250905 19:59:24.241881 29376 consensus_queue.cc:1035] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20250905 19:59:24.268105 28031 debug-util.cc:398] Leaking SignalData structure 0x7b08002e6f20 after lost signal to thread 28109
W20250905 19:59:24.269551 28031 debug-util.cc:398] Leaking SignalData structure 0x7b0800320620 after lost signal to thread 28183
W20250905 19:59:24.270900 28031 debug-util.cc:398] Leaking SignalData structure 0x7b08001942c0 after lost signal to thread 28776
I20250905 19:59:24.401780 29374 raft_consensus.cc:2953] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
W20250905 19:59:24.410135 29371 log.cc:927] Time spent T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.335s user 0.004s sys 0.003s
W20250905 19:59:24.499240 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 57301.6us
I20250905 19:59:24.499419 28072 tablet_service.cc:1515] Processing DeleteTablet for tablet 1b18b7149ff3439a8dde09149cf336b3 with delete_type TABLET_DATA_TOMBSTONED (TS e4413ae5e0b64b52a6f360ac580239b9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59536
I20250905 19:59:24.500048 28010 raft_consensus.cc:2953] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:24.504303 27892 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 5ed920d4c0144d62acf66f595bbf34fd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 19:59:24.645881 29385 tablet_replica.cc:331] T 1b18b7149ff3439a8dde09149cf336b3 P e4413ae5e0b64b52a6f360ac580239b9: stopping tablet replica
I20250905 19:59:24.659233 28082 raft_consensus.cc:1238] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:24.661909 28082 raft_consensus.cc:1273] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:24.666801 28082 raft_consensus.cc:1273] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:24.677359 29337 consensus_queue.cc:1046] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:24.678531 29387 raft_consensus.cc:3053] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:24.678951 29387 raft_consensus.cc:738] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:24.679364 29337 consensus_queue.cc:1035] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:24.679797 29387 consensus_queue.cc:260] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:24.681768 29387 raft_consensus.cc:3058] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:24.682861 29337 consensus_queue.cc:1035] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:24.684952 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:24.701356 28004 raft_consensus.cc:2953] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:24.729149 28003 raft_consensus.cc:2466] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:24.683812 29368 raft_consensus.cc:2953] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:24.765373 29385 raft_consensus.cc:2241] T 1b18b7149ff3439a8dde09149cf336b3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:24.805859 28752 raft_consensus.cc:2953] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:24.813802 28122 leader_election.cc:304] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:24.808867 29385 raft_consensus.cc:2270] T 1b18b7149ff3439a8dde09149cf336b3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:24.855301 29392 raft_consensus.cc:2802] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:24.857446 29392 raft_consensus.cc:695] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:24.858711 29392 consensus_queue.cc:237] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:24.863106 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 49449.4us
W20250905 19:59:24.900108 29394 log.cc:927] Time spent T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20250905 19:59:24.909905 29385 ts_tablet_manager.cc:1905] T 1b18b7149ff3439a8dde09149cf336b3 P e4413ae5e0b64b52a6f360ac580239b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20250905 19:59:24.933193 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 116546us
I20250905 19:59:24.937474 28151 raft_consensus.cc:1273] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:24.941738 28156 raft_consensus.cc:1238] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:24.945832 28151 raft_consensus.cc:1273] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:24.951908 28749 raft_consensus.cc:2953] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:24.969151 27908 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34440 (request call id 224) took 1034 ms. Trace:
I20250905 19:59:24.967926 27905 catalog_manager.cc:5582] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: config changed from index 3 to 4, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:24.971401 27905 catalog_manager.cc:5582] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:24.969522 27908 rpcz_store.cc:276] 0905 19:59:23.935170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:23.935300 (+ 130us) service_pool.cc:225] Handling call
0905 19:59:24.969079 (+1033779us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16921,"reported_tablets":7}
I20250905 19:59:24.980875 28082 raft_consensus.cc:1273] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250905 19:59:24.997375 29173 consensus_queue.cc:1035] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
W20250905 19:59:25.033037 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 98972.2us
I20250905 19:59:25.036269 28153 raft_consensus.cc:1273] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250905 19:59:25.051649 29338 consensus_queue.cc:1035] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20250905 19:59:25.073951 28004 raft_consensus.cc:1273] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250905 19:59:25.061205 29375 consensus_queue.cc:1035] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20250905 19:59:25.063287 29173 raft_consensus.cc:491] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:25.085486 29173 raft_consensus.cc:513] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:25.081146 29337 raft_consensus.cc:491] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:25.089951 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:25.090121 29337 raft_consensus.cc:513] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:25.092792 29337 leader_election.cc:290] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:25.064074 29368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:59536 (request call id 489) took 1263 ms. Trace:
I20250905 19:59:25.094482 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:25.080382 29400 consensus_queue.cc:1035] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 19:59:25.106931 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:25.108529 28122 leader_election.cc:304] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:25.098851 29368 rpcz_store.cc:276] 0905 19:59:23.797532 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:23.797644 (+ 112us) service_pool.cc:225] Handling call
0905 19:59:23.803059 (+ 5415us) log.cc:844] Serialized 501 byte log entry
0905 19:59:25.060510 (+1257451us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2972,"raft.queue_time_us":14153,"raft.run_cpu_time_us":1038,"raft.run_wall_time_us":1039,"thread_start_us":77172,"threads_started":1,"wal-append.queue_time_us":100754,"wal-append.run_cpu_time_us":3841,"wal-append.run_wall_time_us":1002649}
W20250905 19:59:25.111209 27973 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:25.113787 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 54747.2us
I20250905 19:59:25.094638 29173 leader_election.cc:290] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:25.118991 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:25.122195 29402 raft_consensus.cc:2747] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:25.124773 27892 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 761b89d47d5543d89dd47af039cfe7de with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20250905 19:59:25.122684 29406 log.cc:927] Time spent T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.068s user 0.002s sys 0.002s
I20250905 19:59:25.131080 29173 raft_consensus.cc:491] T 5ed920d4c0144d62acf66f595bbf34fd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:25.131767 29173 raft_consensus.cc:513] T 5ed920d4c0144d62acf66f595bbf34fd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:25.135519 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:25.154189 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:25.158540 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:25.156699 28123 leader_election.cc:304] T 5ed920d4c0144d62acf66f595bbf34fd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:25.163151 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:25.161381 27973 leader_election.cc:304] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:25.171996 29388 raft_consensus.cc:2747] T 5ed920d4c0144d62acf66f595bbf34fd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:25.151278 29173 leader_election.cc:290] T 5ed920d4c0144d62acf66f595bbf34fd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:25.167907 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:25.167683 29338 consensus_queue.cc:1035] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20250905 19:59:25.168570 29336 raft_consensus.cc:2747] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250905 19:59:25.200485 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 53820.8us
I20250905 19:59:25.215734 29385 ts_tablet_manager.cc:1918] T 1b18b7149ff3439a8dde09149cf336b3 P e4413ae5e0b64b52a6f360ac580239b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250905 19:59:25.216280 29385 log.cc:1199] T 1b18b7149ff3439a8dde09149cf336b3 P e4413ae5e0b64b52a6f360ac580239b9: Deleting WAL directory at /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-1-root/wals/1b18b7149ff3439a8dde09149cf336b3
I20250905 19:59:25.219538 27892 catalog_manager.cc:4928] TS e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): tablet 1b18b7149ff3439a8dde09149cf336b3 (table test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]) successfully deleted
I20250905 19:59:25.234354 28145 tablet_service.cc:1515] Processing DeleteTablet for tablet 5ed920d4c0144d62acf66f595bbf34fd with delete_type TABLET_DATA_TOMBSTONED (TS 26b2f874b8e54f08b01a9ed00d5cce05 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33742
I20250905 19:59:25.234933 27907 catalog_manager.cc:5582] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 1 to 2, leader changed from cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 2 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20250905 19:59:25.235813 28000 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f356a2522e646889c023d4b3d26f9cc with delete_type TABLET_DATA_TOMBSTONED (TS cd764bfeebac4cc9ad851cd4a6a7d3b0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54144
I20250905 19:59:25.236651 29420 tablet_replica.cc:331] T 5ed920d4c0144d62acf66f595bbf34fd P 26b2f874b8e54f08b01a9ed00d5cce05: stopping tablet replica
I20250905 19:59:25.237872 29420 raft_consensus.cc:2241] T 5ed920d4c0144d62acf66f595bbf34fd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:25.239729 29420 raft_consensus.cc:2270] T 5ed920d4c0144d62acf66f595bbf34fd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:25.249173 27908 catalog_manager.cc:5582] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:25.251485 27908 catalog_manager.cc:5582] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 1 to 2, leader changed from e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 2 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:25.271585 27908 catalog_manager.cc:5582] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 1 to 2, leader changed from e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 2 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:25.305032 28003 raft_consensus.cc:1273] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:25.321679 29420 ts_tablet_manager.cc:1905] T 5ed920d4c0144d62acf66f595bbf34fd P 26b2f874b8e54f08b01a9ed00d5cce05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20250905 19:59:25.307897 28180 debug-util.cc:398] Leaking SignalData structure 0x7b08000833a0 after lost signal to thread 26861
W20250905 19:59:25.326447 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 47493.9us
I20250905 19:59:25.333915 29421 tablet_replica.cc:331] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0: stopping tablet replica
I20250905 19:59:25.335134 29421 raft_consensus.cc:2241] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:25.335933 29421 raft_consensus.cc:2270] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:25.309113 29392 consensus_queue.cc:1035] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
W20250905 19:59:25.341408 28180 debug-util.cc:398] Leaking SignalData structure 0x7b0800170660 after lost signal to thread 27942
W20250905 19:59:25.352375 28180 debug-util.cc:398] Leaking SignalData structure 0x7b080018bae0 after lost signal to thread 28034
W20250905 19:59:25.359520 28180 debug-util.cc:398] Leaking SignalData structure 0x7b08001aaac0 after lost signal to thread 28109
W20250905 19:59:25.363049 28180 debug-util.cc:398] Leaking SignalData structure 0x7b0800330040 after lost signal to thread 28183
W20250905 19:59:25.383149 28180 debug-util.cc:398] Leaking SignalData structure 0x7b080022d480 after lost signal to thread 28776
I20250905 19:59:25.395560 29421 ts_tablet_manager.cc:1905] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 19:59:25.401895 28145 tablet_service.cc:1515] Processing DeleteTablet for tablet 761b89d47d5543d89dd47af039cfe7de with delete_type TABLET_DATA_TOMBSTONED (TS 26b2f874b8e54f08b01a9ed00d5cce05 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33742
I20250905 19:59:25.447280 28082 raft_consensus.cc:1273] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:25.448958 29402 consensus_queue.cc:1035] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20250905 19:59:25.478008 29420 ts_tablet_manager.cc:1918] T 5ed920d4c0144d62acf66f595bbf34fd P 26b2f874b8e54f08b01a9ed00d5cce05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250905 19:59:25.478631 29420 log.cc:1199] T 5ed920d4c0144d62acf66f595bbf34fd P 26b2f874b8e54f08b01a9ed00d5cce05: Deleting WAL directory at /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-2-root/wals/5ed920d4c0144d62acf66f595bbf34fd
I20250905 19:59:25.490918 27894 catalog_manager.cc:4928] TS 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): tablet 5ed920d4c0144d62acf66f595bbf34fd (table test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]) successfully deleted
I20250905 19:59:25.493109 29420 tablet_replica.cc:331] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05: stopping tablet replica
I20250905 19:59:25.500180 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:59:25.504767 29420 raft_consensus.cc:2241] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:25.505730 29420 raft_consensus.cc:2270] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:25.523224 29420 ts_tablet_manager.cc:1905] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 19:59:25.547076 29421 ts_tablet_manager.cc:1918] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250905 19:59:25.550762 29421 log.cc:1199] T 2f356a2522e646889c023d4b3d26f9cc P cd764bfeebac4cc9ad851cd4a6a7d3b0: Deleting WAL directory at /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-0-root/wals/2f356a2522e646889c023d4b3d26f9cc
I20250905 19:59:25.553144 27895 catalog_manager.cc:4928] TS cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): tablet 2f356a2522e646889c023d4b3d26f9cc (table test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]) successfully deleted
I20250905 19:59:25.570396 29420 ts_tablet_manager.cc:1918] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250905 19:59:25.574734 29420 log.cc:1199] T 761b89d47d5543d89dd47af039cfe7de P 26b2f874b8e54f08b01a9ed00d5cce05: Deleting WAL directory at /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-2-root/wals/761b89d47d5543d89dd47af039cfe7de
I20250905 19:59:25.581539 27894 catalog_manager.cc:4928] TS 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): tablet 761b89d47d5543d89dd47af039cfe7de (table test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]) successfully deleted
I20250905 19:59:25.597395 28009 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "0860997b3256400084f5884190bbe6e2"
dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
mode: GRACEFUL
new_leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
from {username='slave'} at 127.0.0.1:47564
I20250905 19:59:25.598110 28009 raft_consensus.cc:604] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Received request to transfer leadership to TS 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:25.607543 28151 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
from {username='slave'} at 127.0.0.1:36164
I20250905 19:59:25.608322 28151 raft_consensus.cc:604] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Received request to transfer leadership to TS 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:25.626197 28151 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:36164
I20250905 19:59:25.626943 28151 raft_consensus.cc:604] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Received request to transfer leadership to TS cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:25.640522 28082 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0"
dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
mode: GRACEFUL
new_leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:51630
I20250905 19:59:25.641506 28082 raft_consensus.cc:604] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Received request to transfer leadership to TS cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:25.645907 28151 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
from {username='slave'} at 127.0.0.1:36164
I20250905 19:59:25.646915 28151 raft_consensus.cc:604] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Received request to transfer leadership to TS 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:25.648301 29402 raft_consensus.cc:991] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05: : Instructing follower 7e9c9f591f05464f9b097bf003d14f4a to start an election
I20250905 19:59:25.648334 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250905 19:59:25.648900 29402 raft_consensus.cc:1079] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Signalling peer 7e9c9f591f05464f9b097bf003d14f4a to start an election
I20250905 19:59:25.650374 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:59:25.652766 28749 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c"
dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
from {username='slave'} at 127.0.0.1:40926
I20250905 19:59:25.653141 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:59:25.653450 28749 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:25.653949 28749 raft_consensus.cc:3058] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:25.703593 28749 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:25.706521 28749 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:25.713703 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:25.714489 28156 raft_consensus.cc:3053] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:25.714859 28156 raft_consensus.cc:738] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:25.715749 28156 consensus_queue.cc:260] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:25.717383 28156 raft_consensus.cc:3058] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:25.720234 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:25.721653 28003 raft_consensus.cc:3058] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:25.727689 28156 raft_consensus.cc:2466] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 2.
I20250905 19:59:25.728144 28003 raft_consensus.cc:2466] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 2.
I20250905 19:59:25.729110 28717 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 19:59:25.729941 29376 raft_consensus.cc:2802] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:25.730552 29376 raft_consensus.cc:695] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 2 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:25.731563 29376 consensus_queue.cc:237] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:25.763382 27907 catalog_manager.cc:5582] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 1 to 2, leader changed from 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) to 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132). New cstate: current_term: 2 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 19:59:25.829950 29336 raft_consensus.cc:991] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: : Instructing follower 26b2f874b8e54f08b01a9ed00d5cce05 to start an election
I20250905 19:59:25.830535 29336 raft_consensus.cc:1079] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Signalling peer 26b2f874b8e54f08b01a9ed00d5cce05 to start an election
I20250905 19:59:25.838649 29400 raft_consensus.cc:991] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05: : Instructing follower cd764bfeebac4cc9ad851cd4a6a7d3b0 to start an election
I20250905 19:59:25.839202 29400 raft_consensus.cc:1079] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Signalling peer cd764bfeebac4cc9ad851cd4a6a7d3b0 to start an election
I20250905 19:59:25.842239 28156 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "0860997b3256400084f5884190bbe6e2"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
from {username='slave'} at 127.0.0.1:33752
I20250905 19:59:25.842971 28156 raft_consensus.cc:491] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:25.843374 28156 raft_consensus.cc:3058] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:25.847446 28009 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d"
dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:54164
I20250905 19:59:25.848163 28009 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:25.848609 28009 raft_consensus.cc:3058] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:25.852486 28156 raft_consensus.cc:513] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:25.863524 28009 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:25.865849 28009 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:25.867276 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:25.868395 28151 raft_consensus.cc:3053] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:25.868731 28151 raft_consensus.cc:738] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:25.869436 28151 consensus_queue.cc:260] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:25.871076 28156 leader_election.cc:290] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:25.873308 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:25.874215 28078 raft_consensus.cc:3058] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:25.875835 28151 raft_consensus.cc:3058] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:25.882004 28078 raft_consensus.cc:2466] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:25.890276 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:25.899190 28082 raft_consensus.cc:3058] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:25.899636 28122 leader_election.cc:304] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:25.900990 29089 raft_consensus.cc:2802] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:25.901490 29089 raft_consensus.cc:695] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:25.901736 28151 raft_consensus.cc:2466] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:59:25.892933 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:25.902868 27975 leader_election.cc:304] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:25.903218 28009 raft_consensus.cc:3053] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:25.902323 29089 consensus_queue.cc:237] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:25.903574 28009 raft_consensus.cc:738] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:25.903999 29308 raft_consensus.cc:2802] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:25.904366 28009 consensus_queue.cc:260] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:25.905849 28009 raft_consensus.cc:3058] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:25.907258 28082 raft_consensus.cc:2466] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:59:25.904718 29308 raft_consensus.cc:695] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:25.909448 29308 consensus_queue.cc:237] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:25.935098 27907 catalog_manager.cc:5582] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 1 to 2, leader changed from cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 2 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20250905 19:59:25.944875 29089 raft_consensus.cc:991] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05: : Instructing follower 7e9c9f591f05464f9b097bf003d14f4a to start an election
I20250905 19:59:25.945423 29089 raft_consensus.cc:1079] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Signalling peer 7e9c9f591f05464f9b097bf003d14f4a to start an election
I20250905 19:59:25.953760 28749 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc"
dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
from {username='slave'} at 127.0.0.1:40926
I20250905 19:59:25.954507 28749 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:25.955201 28749 raft_consensus.cc:3058] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:25.978021 28749 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:25.979933 27907 catalog_manager.cc:5582] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 2 to 3, leader changed from 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 3 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:25.992713 28749 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:25.995707 28009 raft_consensus.cc:2466] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:25.997318 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:25.997987 28082 raft_consensus.cc:3058] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:25.992870 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:25.999166 28151 raft_consensus.cc:3053] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:25.999526 28151 raft_consensus.cc:738] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:26.000349 28151 consensus_queue.cc:260] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:26.001700 28151 raft_consensus.cc:3058] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:26.019516 28151 raft_consensus.cc:2466] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 2.
I20250905 19:59:26.021850 28082 raft_consensus.cc:2466] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 2.
I20250905 19:59:26.028012 28715 leader_election.cc:304] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:26.028964 29376 raft_consensus.cc:2802] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:26.029527 29376 raft_consensus.cc:695] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 2 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:26.030537 29376 consensus_queue.cc:237] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:26.075220 27907 catalog_manager.cc:5582] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 1 to 2, leader changed from 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) to 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132). New cstate: current_term: 2 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 19:59:26.087241 29410 raft_consensus.cc:991] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9: : Instructing follower cd764bfeebac4cc9ad851cd4a6a7d3b0 to start an election
I20250905 19:59:26.087694 29410 raft_consensus.cc:1079] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Signalling peer cd764bfeebac4cc9ad851cd4a6a7d3b0 to start an election
I20250905 19:59:26.092478 28009 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0"
dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:54156
I20250905 19:59:26.093096 28009 raft_consensus.cc:491] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:26.093469 28009 raft_consensus.cc:3058] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:26.099874 28009 raft_consensus.cc:513] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:26.102262 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:26.102972 28151 raft_consensus.cc:3058] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:26.105746 28009 leader_election.cc:290] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:26.102262 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:26.108394 28082 raft_consensus.cc:3053] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:26.108672 28082 raft_consensus.cc:738] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:26.109364 28082 consensus_queue.cc:260] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:26.125895 28082 raft_consensus.cc:3058] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:26.127511 28156 raft_consensus.cc:1273] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250905 19:59:26.129150 29376 consensus_queue.cc:1035] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250905 19:59:26.135205 28151 raft_consensus.cc:2466] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:26.136494 27975 leader_election.cc:304] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:26.137295 29336 raft_consensus.cc:2802] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:26.137796 29336 raft_consensus.cc:695] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:26.139103 29336 consensus_queue.cc:237] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:26.158555 28009 raft_consensus.cc:1273] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250905 19:59:26.162757 29376 consensus_queue.cc:1035] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250905 19:59:26.193853 28082 raft_consensus.cc:2466] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
W20250905 19:59:26.204088 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 50312.5us
I20250905 19:59:26.230981 27907 catalog_manager.cc:5582] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 1 to 2, leader changed from e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 2 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:26.304781 28003 raft_consensus.cc:1273] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:26.306573 29089 consensus_queue.cc:1035] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:26.315212 28082 raft_consensus.cc:1273] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:59:26.328109 29409 consensus_queue.cc:1035] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:26.347329 28078 raft_consensus.cc:1273] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:26.351048 29089 consensus_queue.cc:1035] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:26.400489 28156 consensus_queue.cc:237] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:26.445660 28151 raft_consensus.cc:1273] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:59:26.448874 28080 raft_consensus.cc:1273] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250905 19:59:26.457091 29431 consensus_queue.cc:1035] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250905 19:59:26.458176 28123 consensus_peers.cc:489] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 -> Peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299): Couldn't send request to peer 7e9c9f591f05464f9b097bf003d14f4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 10df9d019f2e4e208f4e23d50f551ad6. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:26.459497 29409 consensus_queue.cc:1035] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:26.465893 28078 raft_consensus.cc:1273] T 10df9d019f2e4e208f4e23d50f551ad6 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:26.467628 29089 consensus_queue.cc:1035] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:26.478684 28007 raft_consensus.cc:1273] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:26.481338 29402 consensus_queue.cc:1035] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:26.491485 28156 raft_consensus.cc:1273] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250905 19:59:26.493049 29376 consensus_queue.cc:1035] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250905 19:59:26.537168 28078 raft_consensus.cc:2953] T 10df9d019f2e4e208f4e23d50f551ad6 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:26.531486 29173 raft_consensus.cc:2953] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
W20250905 19:59:26.585737 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 50005us
W20250905 19:59:26.591841 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 45823.3us
I20250905 19:59:26.625831 28151 consensus_queue.cc:237] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:26.674858 27905 catalog_manager.cc:5582] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250905 19:59:26.684551 28123 consensus_peers.cc:489] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 -> Peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299): Couldn't send request to peer 7e9c9f591f05464f9b097bf003d14f4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc7dc431504743d79dbdc8eea83f9dcb. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:26.687130 28009 raft_consensus.cc:1273] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:26.689389 28080 raft_consensus.cc:1273] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:26.691326 29402 consensus_queue.cc:1035] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:26.702930 29447 consensus_queue.cc:1035] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:26.707480 28007 raft_consensus.cc:2953] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:26.717155 28082 raft_consensus.cc:1273] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:26.719647 29448 consensus_queue.cc:1035] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:26.752812 29173 raft_consensus.cc:2953] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:26.746058 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:59:26.768477 28153 raft_consensus.cc:1273] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:26.776299 29409 consensus_queue.cc:1035] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:26.793797 28003 raft_consensus.cc:2953] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:26.966180 28085 consensus_queue.cc:237] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:26.992570 29464 ts_tablet_manager.cc:927] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a: Initiating tablet copy from peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:27.104543 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 83214.8us
W20250905 19:59:27.105815 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 84076us
W20250905 19:59:27.138206 29465 log.cc:927] Time spent T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.116s user 0.005s sys 0.005s
W20250905 19:59:27.154633 29457 log.cc:927] Time spent T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.198s user 0.004s sys 0.020s
W20250905 19:59:27.164254 28049 consensus_peers.cc:489] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 -> Peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299): Couldn't send request to peer 7e9c9f591f05464f9b097bf003d14f4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 986c4f475a8d46f39465775db70416d3. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:59:27.178844 29460 log.cc:927] Time spent T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.224s user 0.000s sys 0.014s
I20250905 19:59:27.293797 27904 catalog_manager.cc:5582] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250905 19:59:27.303992 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 87788.4us
W20250905 19:59:27.304580 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 154756us
I20250905 19:59:27.311326 28151 raft_consensus.cc:1273] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:27.304580 29464 tablet_copy_client.cc:323] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Beginning tablet copy session from remote peer at address 127.26.58.131:36705
I20250905 19:59:27.327497 28009 raft_consensus.cc:1273] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:59:27.361450 29479 ts_tablet_manager.cc:927] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a: Initiating tablet copy from peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:27.362216 28010 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4"
dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
mode: GRACEFUL
new_leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
from {username='slave'} at 127.0.0.1:47564
I20250905 19:59:27.364308 28010 raft_consensus.cc:604] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Received request to transfer leadership to TS 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:27.377948 28169 tablet_copy_service.cc:140] P 26b2f874b8e54f08b01a9ed00d5cce05: Received BeginTabletCopySession request for tablet 10df9d019f2e4e208f4e23d50f551ad6 from peer 7e9c9f591f05464f9b097bf003d14f4a ({username='slave'} at 127.0.0.1:39586)
I20250905 19:59:27.378643 28169 tablet_copy_service.cc:161] P 26b2f874b8e54f08b01a9ed00d5cce05: Beginning new tablet copy session on tablet 10df9d019f2e4e208f4e23d50f551ad6 from peer 7e9c9f591f05464f9b097bf003d14f4a at {username='slave'} at 127.0.0.1:39586: session id = 7e9c9f591f05464f9b097bf003d14f4a-10df9d019f2e4e208f4e23d50f551ad6
I20250905 19:59:27.397810 29454 consensus_queue.cc:1035] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:27.422669 28169 tablet_copy_source_session.cc:215] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:27.423110 28080 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3"
dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
mode: GRACEFUL
new_leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
from {username='slave'} at 127.0.0.1:51630
I20250905 19:59:27.423313 29377 consensus_queue.cc:1035] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:27.423847 28080 raft_consensus.cc:604] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Received request to transfer leadership to TS 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:27.436749 29464 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10df9d019f2e4e208f4e23d50f551ad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:27.433069 29479 tablet_copy_client.cc:323] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Beginning tablet copy session from remote peer at address 127.26.58.131:36705
I20250905 19:59:27.448814 29377 raft_consensus.cc:991] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9: : Instructing follower 26b2f874b8e54f08b01a9ed00d5cce05 to start an election
I20250905 19:59:27.449424 29454 raft_consensus.cc:1079] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Signalling peer 26b2f874b8e54f08b01a9ed00d5cce05 to start an election
I20250905 19:59:27.456110 28153 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
from {username='slave'} at 127.0.0.1:33756
I20250905 19:59:27.456808 28153 raft_consensus.cc:491] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:27.457201 28153 raft_consensus.cc:3058] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:27.458746 28169 tablet_copy_service.cc:140] P 26b2f874b8e54f08b01a9ed00d5cce05: Received BeginTabletCopySession request for tablet fc7dc431504743d79dbdc8eea83f9dcb from peer 7e9c9f591f05464f9b097bf003d14f4a ({username='slave'} at 127.0.0.1:39586)
I20250905 19:59:27.459837 28169 tablet_copy_service.cc:161] P 26b2f874b8e54f08b01a9ed00d5cce05: Beginning new tablet copy session on tablet fc7dc431504743d79dbdc8eea83f9dcb from peer 7e9c9f591f05464f9b097bf003d14f4a at {username='slave'} at 127.0.0.1:39586: session id = 7e9c9f591f05464f9b097bf003d14f4a-fc7dc431504743d79dbdc8eea83f9dcb
I20250905 19:59:27.461702 28082 raft_consensus.cc:2953] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:27.463429 29469 raft_consensus.cc:2953] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:27.478911 28078 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738"
dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
mode: GRACEFUL
new_leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:51630
I20250905 19:59:27.479698 28078 raft_consensus.cc:604] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Received request to transfer leadership to TS cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:27.482271 28156 raft_consensus.cc:2953] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:27.488744 28004 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8"
dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
mode: GRACEFUL
new_leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
from {username='slave'} at 127.0.0.1:47564
I20250905 19:59:27.489461 28004 raft_consensus.cc:604] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 LEADER]: Received request to transfer leadership to TS 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:27.501247 28155 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
from {username='slave'} at 127.0.0.1:36164
I20250905 19:59:27.501972 28155 raft_consensus.cc:604] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Received request to transfer leadership to TS 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:27.504125 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250905 19:59:27.495115 28153 raft_consensus.cc:513] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:27.511639 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:59:27.512760 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:59:27.534327 28153 leader_election.cc:290] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:27.535760 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:27.536711 28749 raft_consensus.cc:3058] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:27.541234 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:27.542111 28080 raft_consensus.cc:3053] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:27.542518 28080 raft_consensus.cc:738] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:27.543393 28080 consensus_queue.cc:260] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:27.544761 28080 raft_consensus.cc:3058] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:27.563599 28080 raft_consensus.cc:2466] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:27.565331 28122 leader_election.cc:304] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:27.566759 29472 raft_consensus.cc:2802] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:27.567605 29472 raft_consensus.cc:695] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:27.568396 29472 consensus_queue.cc:237] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:27.576361 28169 tablet_copy_source_session.cc:215] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:27.581763 29479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc7dc431504743d79dbdc8eea83f9dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:27.585685 28153 consensus_queue.cc:237] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:27.572857 28009 raft_consensus.cc:2953] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:27.597330 28749 raft_consensus.cc:2466] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:27.620846 28080 raft_consensus.cc:1273] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20250905 19:59:27.621307 28123 consensus_peers.cc:489] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 -> Peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299): Couldn't send request to peer 7e9c9f591f05464f9b097bf003d14f4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e707b0f561934674947fb463ce626a07. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:27.622081 29464 tablet_copy_client.cc:806] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:27.635257 29486 ts_tablet_manager.cc:927] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a: Initiating tablet copy from peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:27.626261 29447 consensus_queue.cc:1035] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20250905 19:59:27.641225 28010 raft_consensus.cc:1273] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250905 19:59:27.644896 29478 consensus_queue.cc:1035] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:27.647066 27905 catalog_manager.cc:5582] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: config changed from index -1 to 2, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:27.648685 29464 tablet_copy_client.cc:670] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:27.663353 29464 tablet_copy_client.cc:538] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20250905 19:59:27.683926 29489 log.cc:927] Time spent T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20250905 19:59:27.690869 29474 raft_consensus.cc:991] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0: : Instructing follower 26b2f874b8e54f08b01a9ed00d5cce05 to start an election
I20250905 19:59:27.691483 29338 raft_consensus.cc:1079] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Signalling peer 26b2f874b8e54f08b01a9ed00d5cce05 to start an election
I20250905 19:59:27.693604 29479 tablet_copy_client.cc:806] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:27.693545 28153 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
from {username='slave'} at 127.0.0.1:33752
I20250905 19:59:27.695158 29338 raft_consensus.cc:991] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0: : Instructing follower 26b2f874b8e54f08b01a9ed00d5cce05 to start an election
I20250905 19:59:27.695699 29338 raft_consensus.cc:1079] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 LEADER]: Signalling peer 26b2f874b8e54f08b01a9ed00d5cce05 to start an election
I20250905 19:59:27.695837 29479 tablet_copy_client.cc:670] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:27.695086 28153 raft_consensus.cc:491] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:27.697552 28153 raft_consensus.cc:3058] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:27.699430 28155 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
from {username='slave'} at 127.0.0.1:33752
I20250905 19:59:27.700026 28155 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:27.700289 28155 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:27.703919 29486 tablet_copy_client.cc:323] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Beginning tablet copy session from remote peer at address 127.26.58.130:34193
I20250905 19:59:27.704589 29479 tablet_copy_client.cc:538] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 19:59:27.710033 27904 catalog_manager.cc:5582] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 1 to 2, leader changed from e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 2 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:27.710335 28155 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:27.720088 28095 tablet_copy_service.cc:140] P e4413ae5e0b64b52a6f360ac580239b9: Received BeginTabletCopySession request for tablet 986c4f475a8d46f39465775db70416d3 from peer 7e9c9f591f05464f9b097bf003d14f4a ({username='slave'} at 127.0.0.1:50660)
I20250905 19:59:27.720773 28095 tablet_copy_service.cc:161] P e4413ae5e0b64b52a6f360ac580239b9: Beginning new tablet copy session on tablet 986c4f475a8d46f39465775db70416d3 from peer 7e9c9f591f05464f9b097bf003d14f4a at {username='slave'} at 127.0.0.1:50660: session id = 7e9c9f591f05464f9b097bf003d14f4a-986c4f475a8d46f39465775db70416d3
I20250905 19:59:27.721864 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:27.722599 28009 raft_consensus.cc:3053] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:59:27.722934 28009 raft_consensus.cc:738] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:27.723703 28009 consensus_queue.cc:260] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:27.724095 28155 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:27.724857 28009 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:27.727823 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:27.728545 28078 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:27.730171 29464 tablet_bootstrap.cc:492] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap starting.
I20250905 19:59:27.742415 29472 raft_consensus.cc:2953] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:27.752846 28010 raft_consensus.cc:2953] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:27.760543 28095 tablet_copy_source_session.cc:215] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:27.766144 29486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 986c4f475a8d46f39465775db70416d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:27.821636 29463 raft_consensus.cc:991] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9: : Instructing follower cd764bfeebac4cc9ad851cd4a6a7d3b0 to start an election
I20250905 19:59:27.822198 29463 raft_consensus.cc:1079] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Signalling peer cd764bfeebac4cc9ad851cd4a6a7d3b0 to start an election
I20250905 19:59:27.863253 28078 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 19:59:27.865935 28122 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:27.874328 28153 raft_consensus.cc:513] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:27.884016 28004 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738"
dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:54156
I20250905 19:59:27.884681 28004 raft_consensus.cc:491] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:27.885052 28004 raft_consensus.cc:3058] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:27.886508 29447 raft_consensus.cc:2802] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:27.894596 28004 raft_consensus.cc:513] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:27.896677 28004 leader_election.cc:290] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:27.906903 28009 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 19:59:27.907447 28153 leader_election.cc:290] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:27.907922 29447 raft_consensus.cc:695] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:27.908215 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:27.908870 28078 raft_consensus.cc:3053] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:27.909217 28078 raft_consensus.cc:738] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:27.909296 29447 consensus_queue.cc:237] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:27.910077 28078 consensus_queue.cc:260] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:27.911321 28078 raft_consensus.cc:3058] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:27.917878 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:27.918607 28004 raft_consensus.cc:3053] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:27.918824 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:27.919605 28082 raft_consensus.cc:3058] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:27.922784 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:27.923398 28155 raft_consensus.cc:3058] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:27.918917 28004 raft_consensus.cc:738] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:27.927799 28004 consensus_queue.cc:260] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:27.952414 29492 log.cc:927] Time spent T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.122s user 0.000s sys 0.003s
I20250905 19:59:27.991142 28082 raft_consensus.cc:2466] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:27.995314 28004 raft_consensus.cc:3058] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
W20250905 19:59:27.999432 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 123139us
W20250905 19:59:28.018412 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 85646.6us
I20250905 19:59:28.033793 28155 raft_consensus.cc:2466] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:28.034798 28004 raft_consensus.cc:2466] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:28.041245 28125 leader_election.cc:304] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:28.042817 27904 catalog_manager.cc:5582] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index -1 to 3, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New cstate: current_term: 2 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
W20250905 19:59:28.070542 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 46362.3us
I20250905 19:59:28.098191 27975 leader_election.cc:304] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:28.100543 29493 raft_consensus.cc:2802] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:28.101076 29493 raft_consensus.cc:695] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
W20250905 19:59:28.112527 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 64237.9us
I20250905 19:59:28.112874 29500 raft_consensus.cc:2802] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election won for term 2
W20250905 19:59:28.117007 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 77697.9us
I20250905 19:59:28.121161 29479 tablet_bootstrap.cc:492] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap starting.
I20250905 19:59:28.115949 29493 consensus_queue.cc:237] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:28.138725 28078 raft_consensus.cc:2466] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:28.149683 29500 raft_consensus.cc:695] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:28.150882 29500 consensus_queue.cc:237] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:28.157305 29402 raft_consensus.cc:991] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05: : Instructing follower 7e9c9f591f05464f9b097bf003d14f4a to start an election
I20250905 19:59:28.157873 29402 raft_consensus.cc:1079] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Signalling peer 7e9c9f591f05464f9b097bf003d14f4a to start an election
I20250905 19:59:28.175499 28749 raft_consensus.cc:1273] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250905 19:59:28.168462 27905 catalog_manager.cc:5582] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 3 to 4, leader changed from cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 4 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
W20250905 19:59:28.224335 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 99667.3us
I20250905 19:59:28.183552 29464 tablet_bootstrap.cc:492] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:28.235584 28749 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd"
dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
from {username='slave'} at 127.0.0.1:40926
I20250905 19:59:28.236122 28749 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:28.236400 28749 raft_consensus.cc:3058] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:28.255471 28082 raft_consensus.cc:2953] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } } }
I20250905 19:59:28.229849 29402 consensus_queue.cc:1035] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
W20250905 19:59:28.279443 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 120135us
I20250905 19:59:28.279238 28749 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:28.285166 28749 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:28.275849 29464 tablet_bootstrap.cc:492] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap complete.
I20250905 19:59:28.303301 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:28.303948 28155 raft_consensus.cc:3053] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:28.304229 28155 raft_consensus.cc:738] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:28.305001 28155 consensus_queue.cc:260] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:28.306105 28155 raft_consensus.cc:3058] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:28.306874 29464 ts_tablet_manager.cc:1397] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent bootstrapping tablet: real 0.577s user 0.031s sys 0.027s
I20250905 19:59:28.297895 29486 tablet_copy_client.cc:806] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:28.322865 29486 tablet_copy_client.cc:670] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:28.333369 29464 raft_consensus.cc:357] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:28.334347 29464 raft_consensus.cc:738] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Initialized, Role: LEARNER
I20250905 19:59:28.340029 29464 consensus_queue.cc:260] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:28.352463 28155 raft_consensus.cc:2466] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 2.
I20250905 19:59:28.348054 29464 ts_tablet_manager.cc:1428] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent starting tablet: real 0.041s user 0.007s sys 0.000s
I20250905 19:59:28.364406 28085 raft_consensus.cc:1273] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250905 19:59:28.373359 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:28.374163 28082 raft_consensus.cc:3058] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:28.378943 28717 leader_election.cc:304] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
W20250905 19:59:28.391408 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 264425us
I20250905 19:59:28.378084 27905 catalog_manager.cc:5582] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 1 to 2, leader changed from cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 2 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
W20250905 19:59:28.402284 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 45888.5us
I20250905 19:59:28.407596 29472 consensus_queue.cc:1035] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250905 19:59:28.402623 29372 raft_consensus.cc:2802] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:28.409514 29372 raft_consensus.cc:695] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 2 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:28.409744 29486 tablet_copy_client.cc:538] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 19:59:28.410605 29372 consensus_queue.cc:237] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:28.404394 27908 catalog_manager.cc:5582] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 1 to 2, leader changed from e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 2 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:28.414618 28080 raft_consensus.cc:1273] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250905 19:59:28.416127 28169 tablet_copy_service.cc:342] P 26b2f874b8e54f08b01a9ed00d5cce05: Request end of tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-10df9d019f2e4e208f4e23d50f551ad6 received from {username='slave'} at 127.0.0.1:39586
I20250905 19:59:28.419195 28169 tablet_copy_service.cc:434] P 26b2f874b8e54f08b01a9ed00d5cce05: ending tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-10df9d019f2e4e208f4e23d50f551ad6 on tablet 10df9d019f2e4e208f4e23d50f551ad6 with peer 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:28.426467 29472 consensus_queue.cc:1035] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:28.432536 28082 raft_consensus.cc:2466] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 2.
I20250905 19:59:28.443456 29464 ts_tablet_manager.cc:927] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a: Initiating tablet copy from peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:28.458556 28009 raft_consensus.cc:1273] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250905 19:59:28.493613 29504 consensus_queue.cc:1035] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20250905 19:59:28.555739 29486 tablet_bootstrap.cc:492] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap starting.
I20250905 19:59:28.562721 29464 tablet_copy_client.cc:323] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Beginning tablet copy session from remote peer at address 127.26.58.131:36705
I20250905 19:59:28.620510 28749 raft_consensus.cc:1215] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250905 19:59:28.641721 28169 tablet_copy_service.cc:140] P 26b2f874b8e54f08b01a9ed00d5cce05: Received BeginTabletCopySession request for tablet e707b0f561934674947fb463ce626a07 from peer 7e9c9f591f05464f9b097bf003d14f4a ({username='slave'} at 127.0.0.1:39586)
I20250905 19:59:28.642638 28169 tablet_copy_service.cc:161] P 26b2f874b8e54f08b01a9ed00d5cce05: Beginning new tablet copy session on tablet e707b0f561934674947fb463ce626a07 from peer 7e9c9f591f05464f9b097bf003d14f4a at {username='slave'} at 127.0.0.1:39586: session id = 7e9c9f591f05464f9b097bf003d14f4a-e707b0f561934674947fb463ce626a07
I20250905 19:59:28.754086 28083 raft_consensus.cc:1273] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:28.756994 28155 raft_consensus.cc:1273] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:28.759686 29494 consensus_queue.cc:1035] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:28.762084 28169 tablet_copy_source_session.cc:215] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 19:59:28.793211 28004 raft_consensus.cc:1273] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:28.815976 29498 consensus_queue.cc:1035] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:28.816815 28080 raft_consensus.cc:1273] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:28.822492 29504 consensus_queue.cc:1035] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:28.772866 29494 consensus_queue.cc:1035] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20250905 19:59:28.769518 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:59:28.861099 29464 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e707b0f561934674947fb463ce626a07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:59:28.923254 27907 catalog_manager.cc:5582] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 1 to 2, leader changed from 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) to 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132). New cstate: current_term: 2 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 19:59:28.939319 28007 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3"
dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
mode: GRACEFUL
new_leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
from {username='slave'} at 127.0.0.1:47564
I20250905 19:59:28.940198 28007 raft_consensus.cc:604] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Received request to transfer leadership to TS 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:28.946813 28155 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "24cf07baf43247b6bf63063106b79836"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:36164
I20250905 19:59:28.947420 28155 raft_consensus.cc:604] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Received request to transfer leadership to TS cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:28.951834 28153 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
from {username='slave'} at 127.0.0.1:36164
I20250905 19:59:28.968088 28153 raft_consensus.cc:604] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Received request to transfer leadership to TS 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:28.975745 29464 tablet_copy_client.cc:806] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 0 data blocks...
I20250905 19:59:28.976603 29464 tablet_copy_client.cc:670] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Starting download of 1 WAL segments...
I20250905 19:59:28.989142 28079 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "a002b516fab044b998f136c341a88b74"
dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
mode: GRACEFUL
new_leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:51630
I20250905 19:59:28.990149 28079 raft_consensus.cc:604] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Received request to transfer leadership to TS cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 19:59:28.988441 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 49939.4us
I20250905 19:59:28.997010 28155 raft_consensus.cc:1273] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250905 19:59:29.000267 29479 tablet_bootstrap.cc:492] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:29.003942 28152 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:36164
I20250905 19:59:29.004635 28152 raft_consensus.cc:604] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Received request to transfer leadership to TS cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:29.015584 28079 raft_consensus.cc:1273] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250905 19:59:29.016461 29522 consensus_queue.cc:1035] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250905 19:59:29.017195 29372 consensus_queue.cc:1035] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250905 19:59:29.019490 29464 tablet_copy_client.cc:538] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 19:59:29.037573 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250905 19:59:29.040683 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250905 19:59:29.038573 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 44112.1us
I20250905 19:59:29.051115 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:59:29.032405 29479 tablet_bootstrap.cc:492] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap complete.
I20250905 19:59:29.074856 29479 ts_tablet_manager.cc:1397] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a: Time spent bootstrapping tablet: real 0.979s user 0.031s sys 0.030s
I20250905 19:59:29.100873 29479 raft_consensus.cc:357] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:29.102108 29479 raft_consensus.cc:738] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Initialized, Role: LEARNER
I20250905 19:59:29.126351 29464 tablet_bootstrap.cc:492] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap starting.
I20250905 19:59:29.150627 29505 raft_consensus.cc:991] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05: : Instructing follower 7e9c9f591f05464f9b097bf003d14f4a to start an election
I20250905 19:59:29.151221 29505 raft_consensus.cc:1079] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Signalling peer 7e9c9f591f05464f9b097bf003d14f4a to start an election
I20250905 19:59:29.155201 29481 raft_consensus.cc:1062] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05: attempting to promote NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a to VOTER
I20250905 19:59:29.159587 28749 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3"
dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
from {username='slave'} at 127.0.0.1:40926
I20250905 19:59:29.160146 28749 raft_consensus.cc:491] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:29.160473 28749 raft_consensus.cc:3058] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:29.168742 29446 raft_consensus.cc:991] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05: : Instructing follower cd764bfeebac4cc9ad851cd4a6a7d3b0 to start an election
I20250905 19:59:29.169313 29446 raft_consensus.cc:1079] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Signalling peer cd764bfeebac4cc9ad851cd4a6a7d3b0 to start an election
I20250905 19:59:29.166267 29481 consensus_queue.cc:237] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:29.177031 28749 raft_consensus.cc:513] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:29.179675 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:29.180334 28080 raft_consensus.cc:3058] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:29.187131 28003 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "24cf07baf43247b6bf63063106b79836"
dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:54164
I20250905 19:59:29.187259 28080 raft_consensus.cc:2466] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 3.
I20250905 19:59:29.187680 28749 leader_election.cc:290] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:29.187819 28003 raft_consensus.cc:491] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:29.188238 28003 raft_consensus.cc:3058] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:29.190542 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:29.145534 29479 consensus_queue.cc:260] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:29.194231 28153 raft_consensus.cc:3053] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:29.194653 28153 raft_consensus.cc:738] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:29.195335 28153 consensus_queue.cc:260] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:29.196421 28153 raft_consensus.cc:3058] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:29.199240 28083 raft_consensus.cc:1238] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:29.205977 28153 raft_consensus.cc:2466] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 3.
I20250905 19:59:29.207011 28717 leader_election.cc:304] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 19:59:29.208961 29510 raft_consensus.cc:2802] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:29.209470 29510 raft_consensus.cc:695] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:29.210379 29510 consensus_queue.cc:237] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:29.212590 28003 raft_consensus.cc:513] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:29.219620 29545 log.cc:927] Time spent T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.069s user 0.001s sys 0.000s
I20250905 19:59:29.222774 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:29.223598 28083 raft_consensus.cc:3058] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:29.240914 29479 ts_tablet_manager.cc:1428] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a: Time spent starting tablet: real 0.166s user 0.007s sys 0.000s
I20250905 19:59:29.253002 28749 raft_consensus.cc:1273] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:29.255491 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:29.256220 28080 raft_consensus.cc:1273] T 10df9d019f2e4e208f4e23d50f551ad6 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:29.256690 28003 leader_election.cc:290] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:29.256543 29504 consensus_queue.cc:1035] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:29.257810 29505 consensus_queue.cc:1035] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:29.259243 28749 raft_consensus.cc:1215] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250905 19:59:29.256309 28153 raft_consensus.cc:3053] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:29.262650 28153 raft_consensus.cc:738] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:29.263804 28153 consensus_queue.cc:260] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:29.264642 29446 raft_consensus.cc:991] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05: : Instructing follower cd764bfeebac4cc9ad851cd4a6a7d3b0 to start an election
I20250905 19:59:29.265317 29446 raft_consensus.cc:1079] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Signalling peer cd764bfeebac4cc9ad851cd4a6a7d3b0 to start an election
I20250905 19:59:29.265388 28153 raft_consensus.cc:3058] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:29.267431 28169 tablet_copy_service.cc:342] P 26b2f874b8e54f08b01a9ed00d5cce05: Request end of tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-fc7dc431504743d79dbdc8eea83f9dcb received from {username='slave'} at 127.0.0.1:39586
I20250905 19:59:29.268081 28169 tablet_copy_service.cc:434] P 26b2f874b8e54f08b01a9ed00d5cce05: ending tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-fc7dc431504743d79dbdc8eea83f9dcb on tablet fc7dc431504743d79dbdc8eea83f9dcb with peer 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:29.270381 28083 raft_consensus.cc:2466] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:29.272151 27973 leader_election.cc:304] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:29.272184 28004 raft_consensus.cc:1273] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:29.273355 29450 raft_consensus.cc:2802] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:29.274395 29450 raft_consensus.cc:695] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:29.275503 29450 consensus_queue.cc:237] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:29.267374 28003 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb"
dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:54164
I20250905 19:59:29.281893 28003 raft_consensus.cc:491] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:29.282492 28003 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:29.288048 29481 consensus_queue.cc:1035] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:29.303586 28003 raft_consensus.cc:513] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:29.309528 29486 tablet_bootstrap.cc:492] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:29.315379 28003 leader_election.cc:290] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:29.335990 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:29.337033 28155 raft_consensus.cc:3053] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:29.337458 28155 raft_consensus.cc:738] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:29.340876 29486 tablet_bootstrap.cc:492] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap complete.
I20250905 19:59:29.341717 29486 ts_tablet_manager.cc:1397] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent bootstrapping tablet: real 0.786s user 0.023s sys 0.015s
I20250905 19:59:29.340133 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:29.343320 28083 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:29.357372 28153 raft_consensus.cc:2466] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:29.392895 29486 raft_consensus.cc:357] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:29.393903 29486 raft_consensus.cc:738] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Initialized, Role: LEARNER
I20250905 19:59:29.412941 29486 consensus_queue.cc:260] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:29.416095 29486 ts_tablet_manager.cc:1428] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent starting tablet: real 0.074s user 0.004s sys 0.004s
I20250905 19:59:29.416615 29446 raft_consensus.cc:2953] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:29.417968 28095 tablet_copy_service.cc:342] P e4413ae5e0b64b52a6f360ac580239b9: Request end of tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-986c4f475a8d46f39465775db70416d3 received from {username='slave'} at 127.0.0.1:50660
I20250905 19:59:29.418488 28095 tablet_copy_service.cc:434] P e4413ae5e0b64b52a6f360ac580239b9: ending tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-986c4f475a8d46f39465775db70416d3 on tablet 986c4f475a8d46f39465775db70416d3 with peer 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:29.424906 29513 raft_consensus.cc:991] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9: : Instructing follower cd764bfeebac4cc9ad851cd4a6a7d3b0 to start an election
I20250905 19:59:29.425591 29463 raft_consensus.cc:1079] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Signalling peer cd764bfeebac4cc9ad851cd4a6a7d3b0 to start an election
I20250905 19:59:29.431316 28009 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "a002b516fab044b998f136c341a88b74"
dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:54156
I20250905 19:59:29.431959 28009 raft_consensus.cc:491] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:29.402287 28083 raft_consensus.cc:2466] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:59:29.436728 28009 raft_consensus.cc:3058] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:29.443204 27973 leader_election.cc:304] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:29.444082 29450 raft_consensus.cc:2802] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:29.445873 29450 raft_consensus.cc:695] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:29.464097 28078 raft_consensus.cc:2953] T 10df9d019f2e4e208f4e23d50f551ad6 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:29.449702 28004 raft_consensus.cc:2953] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:29.470597 29450 consensus_queue.cc:237] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:29.447418 29474 raft_consensus.cc:991] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: : Instructing follower 26b2f874b8e54f08b01a9ed00d5cce05 to start an election
I20250905 19:59:29.473253 29517 raft_consensus.cc:1079] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Signalling peer 26b2f874b8e54f08b01a9ed00d5cce05 to start an election
I20250905 19:59:29.476982 28009 raft_consensus.cc:513] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:29.481756 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:29.482414 28080 raft_consensus.cc:3053] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:29.482793 28080 raft_consensus.cc:738] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:29.484473 27904 catalog_manager.cc:5582] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 1 to 2, leader changed from 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 2 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:29.483585 28080 consensus_queue.cc:260] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:29.488041 28080 raft_consensus.cc:3058] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:29.490964 28009 leader_election.cc:290] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:29.453051 28155 consensus_queue.cc:260] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:29.508020 28152 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
from {username='slave'} at 127.0.0.1:33752
I20250905 19:59:29.508844 28152 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:29.509341 28152 raft_consensus.cc:3058] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:29.511862 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:29.512707 28153 raft_consensus.cc:3058] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:29.514371 28749 raft_consensus.cc:1215] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250905 19:59:29.568722 28155 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:29.582052 28080 raft_consensus.cc:2466] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:29.589416 27905 catalog_manager.cc:5582] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 19:59:29.502182 28752 raft_consensus.cc:2953] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } attrs { replace: true } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:29.599335 28153 raft_consensus.cc:2466] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:29.600991 27975 leader_election.cc:304] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:29.603431 29476 raft_consensus.cc:2802] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:29.603927 29476 raft_consensus.cc:695] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:29.604727 29476 consensus_queue.cc:237] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:29.609457 28152 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:29.619122 27904 catalog_manager.cc:5582] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 2 to 3, leader changed from 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) to 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132). New cstate: current_term: 3 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 19:59:29.620476 28156 consensus_queue.cc:237] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:29.622191 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RunLeaderElection -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 51853us
I20250905 19:59:29.629227 28152 leader_election.cc:290] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:29.633167 28749 raft_consensus.cc:1273] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:29.635285 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:29.635979 28009 raft_consensus.cc:3053] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:29.636241 28009 raft_consensus.cc:738] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:29.640266 28155 raft_consensus.cc:2466] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:59:29.639168 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:29.641274 28009 consensus_queue.cc:260] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:29.654337 28004 raft_consensus.cc:1273] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:29.650259 29464 tablet_bootstrap.cc:492] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:59:29.657701 28747 raft_consensus.cc:3058] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:29.672052 28009 raft_consensus.cc:3058] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:29.671231 28747 raft_consensus.cc:2466] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:29.680088 28123 leader_election.cc:304] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 19:59:29.682494 28009 raft_consensus.cc:2466] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:29.682952 29464 tablet_bootstrap.cc:492] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a: Bootstrap complete.
I20250905 19:59:29.652961 29481 consensus_queue.cc:1035] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20250905 19:59:29.684583 29464 ts_tablet_manager.cc:1397] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent bootstrapping tablet: real 0.559s user 0.025s sys 0.025s
I20250905 19:59:29.685634 28152 raft_consensus.cc:1273] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250905 19:59:29.684784 29481 raft_consensus.cc:2802] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:29.701951 29557 consensus_queue.cc:1035] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20250905 19:59:29.689690 29504 consensus_queue.cc:1035] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20250905 19:59:29.703244 29481 raft_consensus.cc:695] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:29.760190 27908 catalog_manager.cc:5582] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 2 to 3, leader changed from 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 3 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
W20250905 19:59:29.784561 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 83186.9us
I20250905 19:59:29.801703 28082 raft_consensus.cc:1273] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:29.812644 29493 consensus_queue.cc:1035] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:29.795812 29481 consensus_queue.cc:237] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:29.814682 29464 raft_consensus.cc:357] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:29.821929 29464 raft_consensus.cc:738] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Initialized, Role: LEARNER
I20250905 19:59:29.823827 29512 raft_consensus.cc:2953] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:29.827412 29464 consensus_queue.cc:260] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:29.851374 28078 raft_consensus.cc:1273] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250905 19:59:29.869819 29557 consensus_queue.cc:1035] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20250905 19:59:29.881201 27894 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 10df9d019f2e4e208f4e23d50f551ad6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 19:59:29.893076 28152 raft_consensus.cc:1273] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:29.870242 28007 raft_consensus.cc:2953] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:29.908785 27905 catalog_manager.cc:5582] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 1 to 2, leader changed from e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 2 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:29.898304 29464 ts_tablet_manager.cc:1428] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent starting tablet: real 0.213s user 0.009s sys 0.001s
I20250905 19:59:29.930202 28080 raft_consensus.cc:1273] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:59:29.960875 29495 consensus_queue.cc:1035] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20250905 19:59:29.972229 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 64311us
I20250905 19:59:29.983881 29505 raft_consensus.cc:1062] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05: attempting to promote NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a to VOTER
I20250905 19:59:29.987638 28169 tablet_copy_service.cc:342] P 26b2f874b8e54f08b01a9ed00d5cce05: Request end of tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-e707b0f561934674947fb463ce626a07 received from {username='slave'} at 127.0.0.1:39586
I20250905 19:59:29.988278 28169 tablet_copy_service.cc:434] P 26b2f874b8e54f08b01a9ed00d5cce05: ending tablet copy session 7e9c9f591f05464f9b097bf003d14f4a-e707b0f561934674947fb463ce626a07 on tablet e707b0f561934674947fb463ce626a07 with peer 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:29.987847 29505 consensus_queue.cc:237] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:30.002691 28752 raft_consensus.cc:2953] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
W20250905 19:59:30.029932 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 53391.3us
W20250905 19:59:30.030069 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 49253.5us
I20250905 19:59:30.166147 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:59:30.174507 29509 raft_consensus.cc:1062] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9: attempting to promote NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a to VOTER
W20250905 19:59:30.174516 29550 log.cc:927] Time spent T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20250905 19:59:30.178057 29509 consensus_queue.cc:237] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:30.198338 27905 catalog_manager.cc:5582] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: config changed from index 3 to 4, VOTER e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) evicted. New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:30.243716 28151 raft_consensus.cc:1273] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:30.246884 28153 raft_consensus.cc:1273] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:59:30.253458 28752 raft_consensus.cc:1215] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250905 19:59:30.242785 29474 consensus_queue.cc:1035] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20250905 19:59:30.262750 28747 raft_consensus.cc:1273] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:30.274075 28080 raft_consensus.cc:1273] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:30.275758 29446 consensus_queue.cc:1035] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:30.282274 27907 catalog_manager.cc:5582] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 1 to 2, leader changed from cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 2 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:30.292708 28747 raft_consensus.cc:1273] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250905 19:59:30.295713 29566 consensus_queue.cc:1035] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250905 19:59:30.334565 28072 tablet_service.cc:1515] Processing DeleteTablet for tablet 10df9d019f2e4e208f4e23d50f551ad6 with delete_type TABLET_DATA_TOMBSTONED (TS e4413ae5e0b64b52a6f360ac580239b9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59536
I20250905 19:59:30.359709 29476 consensus_queue.cc:1035] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20250905 19:59:30.359651 29581 consensus_queue.cc:1035] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20250905 19:59:30.368584 28083 raft_consensus.cc:1273] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:30.398175 28156 raft_consensus.cc:1273] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:30.400336 29587 tablet_replica.cc:331] T 10df9d019f2e4e208f4e23d50f551ad6 P e4413ae5e0b64b52a6f360ac580239b9: stopping tablet replica
I20250905 19:59:30.402599 29515 consensus_queue.cc:1035] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:30.452454 28751 raft_consensus.cc:1273] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 LEARNER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:30.402109 29587 raft_consensus.cc:2241] T 10df9d019f2e4e208f4e23d50f551ad6 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250905 19:59:30.464824 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 61393.3us
I20250905 19:59:30.470991 29587 raft_consensus.cc:2270] T 10df9d019f2e4e208f4e23d50f551ad6 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:30.403877 29450 consensus_queue.cc:1035] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.054s
I20250905 19:59:30.475865 28007 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582"
dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
mode: GRACEFUL
new_leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
from {username='slave'} at 127.0.0.1:47564
I20250905 19:59:30.479713 28007 raft_consensus.cc:604] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Received request to transfer leadership to TS e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:30.396991 29499 consensus_queue.cc:1035] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.064s
I20250905 19:59:30.481072 28004 raft_consensus.cc:1273] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 19:59:30.507110 28009 raft_consensus.cc:1273] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20250905 19:59:30.508657 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 129209us
W20250905 19:59:30.527704 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 83345.8us
W20250905 19:59:30.541827 29571 log.cc:927] Time spent T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.127s user 0.004s sys 0.000s
W20250905 19:59:30.543232 29589 log.cc:927] Time spent T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
W20250905 19:59:30.543844 29585 log.cc:927] Time spent T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.080s user 0.001s sys 0.000s
I20250905 19:59:30.544739 28004 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42"
dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
mode: GRACEFUL
new_leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
from {username='slave'} at 127.0.0.1:47564
I20250905 19:59:30.545418 28004 raft_consensus.cc:604] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Received request to transfer leadership to TS 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:30.602919 29578 consensus_queue.cc:1035] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
W20250905 19:59:30.562127 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 148756us
I20250905 19:59:30.639643 28007 raft_consensus.cc:1273] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20250905 19:59:30.649214 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 93351.9us
I20250905 19:59:30.687044 29515 consensus_queue.cc:1035] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:30.669075 28078 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e"
dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
mode: GRACEFUL
new_leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
from {username='slave'} at 127.0.0.1:51630
I20250905 19:59:30.731568 28078 raft_consensus.cc:604] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Received request to transfer leadership to TS 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:30.735186 29512 consensus_queue.cc:1035] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20250905 19:59:30.773509 29587 ts_tablet_manager.cc:1905] T 10df9d019f2e4e208f4e23d50f551ad6 P e4413ae5e0b64b52a6f360ac580239b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 19:59:30.778916 29481 consensus_queue.cc:1035] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20250905 19:59:30.795359 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 115706us
W20250905 19:59:30.798815 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 114611us
W20250905 19:59:30.819939 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 90061.4us
I20250905 19:59:30.865105 29515 raft_consensus.cc:991] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9: : Instructing follower 26b2f874b8e54f08b01a9ed00d5cce05 to start an election
I20250905 19:59:30.865677 29515 raft_consensus.cc:1079] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Signalling peer 26b2f874b8e54f08b01a9ed00d5cce05 to start an election
I20250905 19:59:30.865952 29580 raft_consensus.cc:991] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0: : Instructing follower 26b2f874b8e54f08b01a9ed00d5cce05 to start an election
I20250905 19:59:30.866446 29580 raft_consensus.cc:1079] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Signalling peer 26b2f874b8e54f08b01a9ed00d5cce05 to start an election
W20250905 19:59:30.874953 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 114150us
I20250905 19:59:30.917176 28155 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
from {username='slave'} at 127.0.0.1:33756
W20250905 19:59:30.916421 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 263933us
I20250905 19:59:30.918141 28155 raft_consensus.cc:491] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:30.918673 28155 raft_consensus.cc:3058] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:30.971355 28155 raft_consensus.cc:513] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:30.973402 28155 leader_election.cc:290] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:30.975088 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:30.975703 28752 raft_consensus.cc:3058] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:30.994947 28752 raft_consensus.cc:2466] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:31.015071 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250905 19:59:31.018990 28123 leader_election.cc:304] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
W20250905 19:59:31.034521 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 177546us
I20250905 19:59:31.044642 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:59:31.045532 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:59:31.048004 28153 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
from {username='slave'} at 127.0.0.1:33752
I20250905 19:59:31.048636 28153 raft_consensus.cc:491] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:31.048991 28153 raft_consensus.cc:3058] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
W20250905 19:59:31.097241 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 47393.2us
W20250905 19:59:31.131130 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 243600us
I20250905 19:59:31.168794 28153 raft_consensus.cc:513] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:31.163980 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 57813.8us
I20250905 19:59:31.173283 28153 leader_election.cc:290] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:31.175597 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:31.176474 28752 raft_consensus.cc:3058] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:31.184676 28154 raft_consensus.cc:1238] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
W20250905 19:59:31.190244 29611 log.cc:927] Time spent T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.077s user 0.000s sys 0.001s
I20250905 19:59:31.206347 28752 raft_consensus.cc:2466] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:31.207443 28123 leader_election.cc:304] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 19:59:31.213941 29616 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:31.214761 29616 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
W20250905 19:59:31.217873 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 44736.1us
I20250905 19:59:31.226667 29568 raft_consensus.cc:2802] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:31.231253 29568 raft_consensus.cc:695] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:31.231611 29623 raft_consensus.cc:2802] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:31.287946 29493 raft_consensus.cc:991] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0: : Instructing follower e4413ae5e0b64b52a6f360ac580239b9 to start an election
I20250905 19:59:31.288563 29493 raft_consensus.cc:1079] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Signalling peer e4413ae5e0b64b52a6f360ac580239b9 to start an election
I20250905 19:59:31.243854 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:31.294600 29623 raft_consensus.cc:695] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:31.299274 29509 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:31.310894 29512 raft_consensus.cc:1062] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05: attempting to promote NON_VOTER 7e9c9f591f05464f9b097bf003d14f4a to VOTER
I20250905 19:59:31.327077 29628 consensus_queue.cc:1046] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.078s
I20250905 19:59:31.334928 29616 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:31.349884 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:31.357164 28083 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582"
dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
from {username='slave'} at 127.0.0.1:59550
I20250905 19:59:31.357990 28083 raft_consensus.cc:491] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:31.358652 28083 raft_consensus.cc:3058] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:31.329118 29568 consensus_queue.cc:237] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:31.372291 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 63319.8us
W20250905 19:59:31.371546 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 200418us
I20250905 19:59:31.324659 29509 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:31.394037 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:31.397588 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:31.404004 28002 raft_consensus.cc:3053] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:31.404345 28002 raft_consensus.cc:738] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:31.405071 28002 consensus_queue.cc:260] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:31.310681 29623 consensus_queue.cc:237] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:31.406044 28002 raft_consensus.cc:3058] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:31.414126 28083 raft_consensus.cc:513] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:31.425334 28083 leader_election.cc:290] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:31.428931 28082 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:59:31.434209 29561 raft_consensus.cc:491] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:31.451946 29627 raft_consensus.cc:2953] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
W20250905 19:59:31.477052 27973 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:31.477011 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:31.509758 29509 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:31.506345 28002 raft_consensus.cc:2466] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:31.512827 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:59:31.532727 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 224147us
I20250905 19:59:31.515566 28747 raft_consensus.cc:3058] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:31.550029 29587 ts_tablet_manager.cc:1918] T 10df9d019f2e4e208f4e23d50f551ad6 P e4413ae5e0b64b52a6f360ac580239b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250905 19:59:31.561218 29587 log.cc:1199] T 10df9d019f2e4e208f4e23d50f551ad6 P e4413ae5e0b64b52a6f360ac580239b9: Deleting WAL directory at /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-1-root/wals/10df9d019f2e4e208f4e23d50f551ad6
I20250905 19:59:31.564553 29512 consensus_queue.cc:237] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:31.466650 29561 raft_consensus.cc:513] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:31.577455 29633 raft_consensus.cc:491] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:31.578115 29633 raft_consensus.cc:513] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:31.592335 28078 raft_consensus.cc:3053] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:31.592921 28078 raft_consensus.cc:738] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:31.593820 28078 consensus_queue.cc:260] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:31.587570 29561 leader_election.cc:290] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:31.627125 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:31.627807 28003 raft_consensus.cc:3053] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:31.628126 28003 raft_consensus.cc:738] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:31.635762 28003 consensus_queue.cc:260] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:31.645399 28003 raft_consensus.cc:3058] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:31.645830 29633 leader_election.cc:290] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:31.650156 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:31.606508 29587 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59536 (request call id 519) took 1272 ms. Trace:
W20250905 19:59:31.652895 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 285200us
I20250905 19:59:31.653244 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:31.649834 28009 raft_consensus.cc:2953] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:31.660691 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:31.665653 28003 raft_consensus.cc:2466] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
W20250905 19:59:31.723031 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:31.658497 29587 rpcz_store.cc:276] 0905 19:59:30.324686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:30.334246 (+ 9560us) service_pool.cc:225] Handling call
0905 19:59:30.399824 (+ 65578us) ts_tablet_manager.cc:1145] Deleting tablet 10df9d019f2e4e208f4e23d50f551ad6
0905 19:59:30.399982 (+ 158us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:59:30.630709 (+230727us) spinlock_profiling.cc:243] Waited 2.93 ms on lock 0x7b240011f1f8. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49aaf9784 00007fe49aaf788b 00007fe49ab86669 00007fe49ab862f5 00007fe49ab85b2b 00007fe49d85a73b 00007fe49d85ee13 00007fe4a1600465 00007fe4a2bcd41a 00007fe4a2bfd270
0905 19:59:31.286225 (+655516us) tablet_metadata.cc:641] Metadata flushed
0905 19:59:31.511443 (+225218us) tablet_metadata.cc:641] Metadata flushed
0905 19:59:31.596800 (+ 85357us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2829,"spinlock_wait_cycles":6732544,"tablet-delete.queue_time_us":64576,"thread_start_us":62336,"threads_started":1}
W20250905 19:59:31.736735 27973 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:31.769395 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 59091.2us
W20250905 19:59:31.769336 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 68874.2us
W20250905 19:59:31.876734 29640 log.cc:927] Time spent T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.244s user 0.001s sys 0.000s
I20250905 19:59:31.938510 29509 raft_consensus.cc:491] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:32.018030 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:59:32.029367 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 323855us
I20250905 19:59:32.060286 28155 raft_consensus.cc:1238] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:32.062664 27892 catalog_manager.cc:4928] TS e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): tablet 10df9d019f2e4e208f4e23d50f551ad6 (table test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]) successfully deleted
I20250905 19:59:32.049446 29639 raft_consensus.cc:491] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:32.067023 29639 raft_consensus.cc:513] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:32.093107 29627 consensus_meta.cc:287] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.636s user 0.009s sys 0.002s
I20250905 19:59:32.094812 28718 leader_election.cc:304] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:32.096391 29560 raft_consensus.cc:2747] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:32.049872 28747 raft_consensus.cc:2466] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
W20250905 19:59:32.162741 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 117577us
I20250905 19:59:32.053326 29509 raft_consensus.cc:513] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:32.415591 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 59511.7us
I20250905 19:59:32.425376 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:32.428902 29627 raft_consensus.cc:491] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:32.441731 29627 raft_consensus.cc:513] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:32.452108 29627 leader_election.cc:290] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:32.462291 28750 raft_consensus.cc:1273] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250905 19:59:32.466023 29599 consensus_queue.cc:1035] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
W20250905 19:59:32.522631 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:32.525061 28155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1262) took 1141 ms. Trace:
I20250905 19:59:32.525585 28155 rpcz_store.cc:276] 0905 19:59:31.379840 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:31.380021 (+ 181us) service_pool.cc:225] Handling call
0905 19:59:31.380377 (+ 356us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:32.060781 (+680404us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:32.521257 (+460476us) spinlock_profiling.cc:243] Waited 680 ms on lock 0x7b540009ddd4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:32.521440 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1563147008}
I20250905 19:59:32.486833 29509 leader_election.cc:290] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:32.539918 29617 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:32.522241 29590 raft_consensus.cc:491] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:32.527218 29599 raft_consensus.cc:491] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:32.526959 29499 raft_consensus.cc:491] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:32.540582 29617 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:32.568819 29653 log.cc:927] Time spent T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20250905 19:59:32.547602 29590 raft_consensus.cc:513] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:32.588989 28080 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.710s user 0.017s sys 0.027s
W20250905 19:59:32.589425 28080 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.711s user 0.017s sys 0.027s
I20250905 19:59:32.597581 29590 leader_election.cc:290] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:32.601167 29648 log.cc:927] Time spent T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.062s user 0.000s sys 0.001s
I20250905 19:59:32.594727 28080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1194) took 1164 ms. Trace:
I20250905 19:59:32.549849 29599 raft_consensus.cc:513] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:32.551155 29499 raft_consensus.cc:513] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:32.591171 29512 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.849s user 0.015s sys 0.023s
I20250905 19:59:32.603895 28080 rpcz_store.cc:276] 0905 19:59:31.426385 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:31.426583 (+ 198us) service_pool.cc:225] Handling call
0905 19:59:31.426927 (+ 344us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:31.427120 (+ 193us) raft_consensus.cc:1532] Early marking committed up to index 3
0905 19:59:31.878387 (+451267us) log.cc:844] Serialized 14 byte log entry
0905 19:59:32.590209 (+711822us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:32.590279 (+ 70us) raft_consensus.cc:1643] Marking committed up to 3
0905 19:59:32.590325 (+ 46us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:32.590391 (+ 66us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:32.590864 (+ 473us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":158,"prepare.queue_time_us":999487,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":11119,"raft.queue_time_us":124,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":100,"thread_start_us":1154564,"threads_started":2,"wal-append.queue_time_us":710722}
W20250905 19:59:32.614187 29512 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.890s user 0.016s sys 0.023s
I20250905 19:59:32.639555 29628 raft_consensus.cc:491] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:32.640232 29628 raft_consensus.cc:513] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:32.644675 29505 raft_consensus.cc:491] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:32.647203 29599 leader_election.cc:290] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:32.647396 29499 leader_election.cc:290] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:32.647143 29505 raft_consensus.cc:513] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:32.646261 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 54579.2us
I20250905 19:59:32.649662 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:32.650102 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:32.652084 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:32.676203 29505 leader_election.cc:290] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:32.679407 28078 raft_consensus.cc:3058] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:32.679811 28049 leader_election.cc:304] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:32.682679 29617 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:32.684536 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:32.687887 29508 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:32.725450 29660 raft_consensus.cc:491] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
W20250905 19:59:32.729125 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 78095.9us
W20250905 19:59:32.732293 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 79790.9us
I20250905 19:59:32.645704 29628 leader_election.cc:290] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:32.734197 29622 raft_consensus.cc:2802] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:32.749109 28153 raft_consensus.cc:1238] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:32.757036 28752 raft_consensus.cc:2953] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:32.746706 28078 raft_consensus.cc:2466] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:32.738198 29450 raft_consensus.cc:491] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
W20250905 19:59:32.766886 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 110838us
I20250905 19:59:32.739073 29599 raft_consensus.cc:491] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:32.726317 29660 raft_consensus.cc:513] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:32.741519 29627 raft_consensus.cc:491] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
W20250905 19:59:32.806943 28078 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1188) took 1539 ms (client timeout 1662 ms). Trace:
W20250905 19:59:32.883072 29507 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
W20250905 19:59:32.884709 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:32.688760 29508 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:32.890738 28078 rpcz_store.cc:269] 0905 19:59:31.224626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:31.230021 (+ 5395us) service_pool.cc:225] Handling call
0905 19:59:31.592160 (+362139us) spinlock_profiling.cc:243] Waited 73.3 ms on lock 0x7b54002f0e20. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dba8550 00007fe49dc1cdc8 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216
0905 19:59:32.679126 (+1086966us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b580002a2b0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49db94003 00007fe49dbf7577 00007fe49dc0ffa7 00007fe49dc1179e 00007fe49dc1d22c 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc
0905 19:59:32.763781 (+ 84655us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2538944128}
I20250905 19:59:32.895747 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:32.803499 29626 raft_consensus.cc:491] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:32.884681 29617 raft_consensus.cc:491] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:32.906665 28752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40918 (request call id 343) took 1441 ms. Trace:
I20250905 19:59:32.906913 29597 raft_consensus.cc:491] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:32.907528 29597 raft_consensus.cc:513] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:32.907202 28752 rpcz_store.cc:276] 0905 19:59:31.459558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:31.459740 (+ 182us) service_pool.cc:225] Handling call
0905 19:59:32.756239 (+1296499us) spinlock_profiling.cc:243] Waited 6.97 ms on lock 0x7b5800304598. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b91f60 00007fe4a2b7d1a9 00007fe4a2b7cc52 00007fe49c202b44
0905 19:59:32.756537 (+ 298us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:32.756842 (+ 305us) raft_consensus.cc:1532] Early marking committed up to index 3
0905 19:59:32.887180 (+130338us) log.cc:844] Serialized 14 byte log entry
0905 19:59:32.900597 (+ 13417us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:32.900757 (+ 160us) raft_consensus.cc:1643] Marking committed up to 3
0905 19:59:32.900793 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:32.900843 (+ 50us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:32.900932 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":87,"spinlock_wait_cycles":16022912,"thread_start_us":13082,"threads_started":1}
I20250905 19:59:32.907058 29665 raft_consensus.cc:2953] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
W20250905 19:59:32.910354 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:32.887274 29609 raft_consensus.cc:491] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:32.893038 29578 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:32.914983 29574 raft_consensus.cc:491] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
W20250905 19:59:32.900306 28122 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.662s (SENT) [suppressed 3 similar messages]
I20250905 19:59:32.884521 29450 raft_consensus.cc:513] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 19:59:32.835481 29599 raft_consensus.cc:513] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:32.901121 29622 raft_consensus.cc:695] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:32.909850 29597 leader_election.cc:290] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:32.891698 29627 raft_consensus.cc:513] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:32.899274 29623 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:32.940682 29509 raft_consensus.cc:491] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:32.937165 29617 raft_consensus.cc:513] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:32.962139 29624 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:32.959985 28085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1195) took 1323 ms. Trace:
I20250905 19:59:32.966795 28745 raft_consensus.cc:1238] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:32.966946 29450 leader_election.cc:290] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:32.964915 28085 rpcz_store.cc:276] 0905 19:59:31.636966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:31.637075 (+ 109us) service_pool.cc:225] Handling call
0905 19:59:32.065724 (+428649us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:32.066148 (+ 424us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:32.066179 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:32.351167 (+284988us) log.cc:844] Serialized 564 byte log entry
0905 19:59:32.636948 (+285781us) raft_consensus.cc:1643] Marking committed up to 2
0905 19:59:32.636978 (+ 30us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:32.637035 (+ 57us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:32.651097 (+ 14062us) raft_consensus.cc:1683] finished
0905 19:59:32.651121 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:32.954961 (+303840us) spinlock_profiling.cc:243] Waited 428 ms on lock 0x7b540036db50. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:32.959855 (+ 4894us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":984189184,"thread_start_us":268211,"threads_started":1,"wal-append.queue_time_us":292306}
I20250905 19:59:32.950834 29619 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:32.967880 28154 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:33742 (request call id 617) took 1324 ms. Trace:
I20250905 19:59:32.971243 28154 rpcz_store.cc:276] 0905 19:59:31.640343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:31.640523 (+ 180us) service_pool.cc:225] Handling call
0905 19:59:32.963603 (+1323080us) spinlock_profiling.cc:243] Waited 995 ms on lock 0x7b5400073ad4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9dcd0 00007fe49dc21eea 00007fe4a2b8161e 00007fe49c217cc7 00007fe49c217c52 00007fe49c217b81 00007fe49c217afc 00007fe49c216db2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:32.964504 (+ 901us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2287588224}
I20250905 19:59:32.974851 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:32.913266 29626 raft_consensus.cc:513] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:32.972157 29604 raft_consensus.cc:491] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:32.965924 29509 raft_consensus.cc:513] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:32.944860 29578 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:32.960484 29618 raft_consensus.cc:491] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:32.918484 29609 raft_consensus.cc:513] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:32.962783 29624 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:32.915911 29574 raft_consensus.cc:513] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:33.019706 29599 leader_election.cc:290] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:33.010051 29619 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:33.073504 29619 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:32.975839 29644 raft_consensus.cc:491] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:32.971661 29450 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:33.086392 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 19:59:33.091368 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 191632us
I20250905 19:59:33.018399 29660 leader_election.cc:290] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:32.976116 29596 raft_consensus.cc:491] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:33.054888 29666 raft_consensus.cc:491] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:33.095993 29609 leader_election.cc:290] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:32.965746 29623 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:33.100589 29624 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:33.102406 29599 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:33.098194 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:33.019706 29508 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:33.019861 29668 raft_consensus.cc:491] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:33.087985 29627 leader_election.cc:290] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:33.112716 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 49012.3us
I20250905 19:59:33.091161 29450 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:33.088630 29604 raft_consensus.cc:513] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:33.014526 29622 consensus_queue.cc:237] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:33.115139 29617 leader_election.cc:290] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:33.088236 29618 raft_consensus.cc:513] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:33.123498 29658 raft_consensus.cc:491] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:33.114056 29599 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:33.198789 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 89965.9us
I20250905 19:59:33.184892 29629 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:33.181797 29627 raft_consensus.cc:491] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:33.121903 29666 raft_consensus.cc:513] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:33.213212 29512 raft_consensus.cc:491] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
W20250905 19:59:33.218084 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:33.214093 29505 raft_consensus.cc:491] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:33.188185 29597 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:33.195712 29671 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:33.196182 29508 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:33.213642 29604 leader_election.cc:290] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:33.215793 29580 raft_consensus.cc:491] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:33.115432 29596 raft_consensus.cc:513] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:33.112793 29644 raft_consensus.cc:513] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:33.124486 29668 raft_consensus.cc:513] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:33.222384 29505 raft_consensus.cc:513] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:33.219898 29627 raft_consensus.cc:513] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:33.231724 27974 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:33.220237 29512 raft_consensus.cc:513] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:33.018195 28745 raft_consensus.cc:2953] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:33.221911 29629 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:33.302075 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:33.239527 28082 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1200) took 1190 ms. Trace:
I20250905 19:59:33.228817 29508 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:33.240229 29578 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:33.279874 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:33.306981 29678 raft_consensus.cc:491] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:33.303054 28082 rpcz_store.cc:276] 0905 19:59:32.049544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:33.238650 (+1189106us) service_pool.cc:225] Handling call
0905 19:59:33.239027 (+ 377us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:33.239203 (+ 176us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:33.239226 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:33.239266 (+ 40us) raft_consensus.cc:1643] Marking committed up to 1
0905 19:59:33.239294 (+ 28us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:33.239344 (+ 50us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:33.239468 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:33.296106 29604 raft_consensus.cc:491] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:33.265872 29562 raft_consensus.cc:491] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:33.266084 29509 leader_election.cc:290] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:33.300689 29505 leader_election.cc:290] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:33.300300 29623 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:33.295320 29626 leader_election.cc:290] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:33.239929 29645 raft_consensus.cc:491] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:33.319985 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:33.229009 29597 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:33.327034 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 97924.6us
I20250905 19:59:33.313170 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:33.229135 29675 raft_consensus.cc:491] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:33.216627 29658 raft_consensus.cc:513] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:33.326035 28082 raft_consensus.cc:2466] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:33.328925 28084 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1199) took 1276 ms. Trace:
I20250905 19:59:33.328974 29517 raft_consensus.cc:2747] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:33.328564 29669 raft_consensus.cc:491] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:33.317559 29624 raft_consensus.cc:491] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:33.230392 29671 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:33.322471 29678 raft_consensus.cc:513] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:33.365029 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:33.383575 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:33.236608 29580 raft_consensus.cc:513] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:33.321280 29604 raft_consensus.cc:513] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:33.335397 28084 rpcz_store.cc:276] 0905 19:59:32.051323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:33.321624 (+1270301us) service_pool.cc:225] Handling call
0905 19:59:33.326863 (+ 5239us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:33.327078 (+ 215us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:33.327108 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:33.327151 (+ 43us) raft_consensus.cc:1643] Marking committed up to 2
0905 19:59:33.327180 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:33.327245 (+ 65us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:33.327346 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:33.469111 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:33.314839 29574 leader_election.cc:290] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:33.497296 28080 raft_consensus.cc:2466] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:33.325497 29617 raft_consensus.cc:491] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:33.330715 29626 raft_consensus.cc:491] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:33.475338 29599 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:33.421381 29676 raft_consensus.cc:491] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:33.559326 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:33.354915 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:33.398640 29645 raft_consensus.cc:513] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:33.400586 29662 raft_consensus.cc:491] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:33.598915 29688 raft_consensus.cc:491] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:33.546984 29680 raft_consensus.cc:491] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:33.412971 29512 leader_election.cc:290] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:33.425729 29450 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:33.573613 29682 raft_consensus.cc:491] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:33.560791 29677 raft_consensus.cc:491] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:33.477787 29668 leader_election.cc:290] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:33.503010 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:33.627043 28158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1277) took 1461 ms. Trace:
I20250905 19:59:33.521719 29624 raft_consensus.cc:513] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:33.558378 29666 leader_election.cc:290] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:33.547329 29669 raft_consensus.cc:513] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:33.426522 29618 leader_election.cc:290] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:33.354911 29644 leader_election.cc:290] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:33.607280 28153 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 1760) took 2109 ms. Trace:
I20250905 19:59:33.625784 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:33.654398 28153 rpcz_store.cc:276] 0905 19:59:31.479472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:31.479637 (+ 165us) service_pool.cc:225] Handling call
0905 19:59:31.480026 (+ 389us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:32.755594 (+1275568us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:33.575017 (+819423us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b5400186d54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:33.589006 (+ 13989us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2917944704}
I20250905 19:59:33.550734 29574 raft_consensus.cc:491] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:33.561869 29626 raft_consensus.cc:513] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:33.559808 29617 raft_consensus.cc:513] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:33.428707 29596 leader_election.cc:290] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:33.425086 29681 raft_consensus.cc:491] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:33.503466 29623 raft_consensus.cc:491] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:33.653914 28081 raft_consensus.cc:2374] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2. Re-sending same reply.
I20250905 19:59:33.644042 29627 leader_election.cc:290] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:33.606632 29645 leader_election.cc:290] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:33.641676 29508 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:33.641654 29682 raft_consensus.cc:513] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:33.599222 29662 raft_consensus.cc:513] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:33.734270 28080 raft_consensus.cc:2466] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
W20250905 19:59:33.739315 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:33.734004 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:33.653657 29671 leader_election.cc:290] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:33.741034 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:33.756395 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:33.354768 29675 raft_consensus.cc:513] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:33.326486 29562 raft_consensus.cc:513] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:33.778121 28078 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:59:33.600260 29680 raft_consensus.cc:513] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:33.645115 29677 raft_consensus.cc:513] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:33.599586 29688 raft_consensus.cc:513] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:33.666210 29666 raft_consensus.cc:491] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:33.672629 29580 leader_election.cc:290] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:33.910899 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:33.826876 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:33.807384 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:33.805274 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:59:33.937672 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:33.687989 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:33.976595 28076 raft_consensus.cc:2466] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:33.976680 29687 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:33.977077 29568 raft_consensus.cc:491] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:33.977348 29687 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:33.977650 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:33.977613 29568 raft_consensus.cc:513] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:33.979564 29675 leader_election.cc:290] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:33.649252 29676 raft_consensus.cc:513] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:33.801461 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:33.980088 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:33.688892 29596 raft_consensus.cc:491] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:33.974123 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:33.977771 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:33.981343 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:33.982935 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:33.778764 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:33.997460 28750 raft_consensus.cc:2466] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:59:33.891590 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:59:33.795436 29684 log.cc:927] Time spent T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.221s user 0.000s sys 0.000s
W20250905 19:59:33.796664 29665 consensus_meta.cc:287] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.880s user 0.012s sys 0.000s
I20250905 19:59:33.988253 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:33.937203 28081 raft_consensus.cc:2466] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:34.003786 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:33.817397 29618 raft_consensus.cc:491] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:34.007818 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:33.871652 29639 leader_election.cc:290] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:34.008206 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:34.005086 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:34.000262 28010 raft_consensus.cc:2466] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 19:59:33.742918 29597 leader_election.cc:290] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:33.998865 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:34.016446 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:33.750973 29678 leader_election.cc:290] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:34.007859 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:33.793036 28747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40918 (request call id 347) took 1375 ms. Trace:
I20250905 19:59:33.778559 29685 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:33.797935 29624 leader_election.cc:290] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:34.023057 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:33.864357 29604 leader_election.cc:290] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:34.012293 28006 raft_consensus.cc:1273] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250905 19:59:34.012331 29599 raft_consensus.cc:491] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:34.012795 29682 leader_election.cc:290] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:34.012431 29638 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:34.040978 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:34.041724 29624 raft_consensus.cc:491] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:34.015156 29626 leader_election.cc:290] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:33.892083 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:33.797741 29635 raft_consensus.cc:491] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:33.983980 29644 raft_consensus.cc:491] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:34.020567 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:33.835943 29629 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:33.966095 29590 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:34.049841 29644 raft_consensus.cc:513] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:33.979593 29687 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:33.688443 29574 raft_consensus.cc:513] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:34.021836 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 19:59:34.017874 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 91715.2us
I20250905 19:59:33.888660 29512 raft_consensus.cc:491] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:34.025139 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:33.831419 29658 leader_election.cc:290] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:33.636875 28158 rpcz_store.cc:276] 0905 19:59:32.165347 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:32.166859 (+ 1512us) service_pool.cc:225] Handling call
0905 19:59:33.625961 (+1459102us) spinlock_profiling.cc:243] Waited 6.27 ms on lock 0x7b580015d598. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b91f60 00007fe4a2b7d1a9 00007fe4a2b7cc52 00007fe49c202b44
0905 19:59:33.626240 (+ 279us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:33.626489 (+ 249us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:33.626523 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:33.626585 (+ 62us) raft_consensus.cc:1643] Marking committed up to 3
0905 19:59:33.626624 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:33.626695 (+ 71us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:33.626860 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14425984}
I20250905 19:59:33.893275 29669 leader_election.cc:290] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:34.041625 29638 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:34.082660 29638 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:34.040313 29501 raft_consensus.cc:491] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:34.083684 29501 raft_consensus.cc:513] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:34.040933 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:34.034058 27974 leader_election.cc:304] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:33.749786 29623 raft_consensus.cc:513] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 19:59:34.045886 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:34.095705 28010 raft_consensus.cc:2466] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 1.
I20250905 19:59:34.031584 29513 raft_consensus.cc:491] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:34.053544 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:34.020684 29622 raft_consensus.cc:491] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:34.050706 29590 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:34.016842 28744 raft_consensus.cc:2466] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:34.006076 29596 raft_consensus.cc:513] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:34.002144 29666 raft_consensus.cc:513] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:34.061214 29645 raft_consensus.cc:491] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:34.064741 29644 leader_election.cc:290] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:34.066056 29574 leader_election.cc:290] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:34.067165 28752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40926 (request call id 304) took 1102 ms. Trace:
I20250905 19:59:33.797875 29681 raft_consensus.cc:513] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:34.066598 29512 raft_consensus.cc:513] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:34.032039 29618 raft_consensus.cc:513] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:34.040987 29599 raft_consensus.cc:513] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:34.104255 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 63229.7us
W20250905 19:59:34.104959 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 70964.6us
W20250905 19:59:34.104962 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 96034.8us
I20250905 19:59:34.058805 29662 leader_election.cc:290] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:34.110792 29514 raft_consensus.cc:491] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:34.110908 29645 raft_consensus.cc:513] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:34.114094 29596 leader_election.cc:290] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:34.115778 28752 rpcz_store.cc:276] 0905 19:59:32.964980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:32.965132 (+ 152us) service_pool.cc:225] Handling call
0905 19:59:32.968002 (+ 2870us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:32.968219 (+ 217us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:32.968237 (+ 18us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:32.970905 (+ 2668us) log.cc:844] Serialized 26 byte log entry
0905 19:59:33.087207 (+116302us) raft_consensus.cc:1643] Marking committed up to 4
0905 19:59:33.087247 (+ 40us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:33.087330 (+ 83us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:33.297374 (+210044us) raft_consensus.cc:1683] finished
0905 19:59:33.297405 (+ 31us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:34.053796 (+756391us) spinlock_profiling.cc:243] Waited 2.52 ms on lock 0x7b540053cd50. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:34.067080 (+ 13284us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5797760,"thread_start_us":68214,"threads_started":1,"wal-append.queue_time_us":129343}
I20250905 19:59:34.140724 27907 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34446 (request call id 236) took 2574 ms. Trace:
I20250905 19:59:34.116891 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:34.141072 27974 leader_election.cc:304] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:34.118597 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:34.119531 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:34.087178 29691 raft_consensus.cc:491] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:34.051396 29635 raft_consensus.cc:513] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:34.053648 29624 raft_consensus.cc:513] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:34.087170 29605 raft_consensus.cc:491] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:34.102224 29501 leader_election.cc:290] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:34.098645 29513 raft_consensus.cc:513] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:34.020143 28747 rpcz_store.cc:276] 0905 19:59:32.417877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:32.419411 (+ 1534us) service_pool.cc:225] Handling call
0905 19:59:33.792175 (+1372764us) spinlock_profiling.cc:243] Waited 21.8 ms on lock 0x7b5800304598. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac6721e 000000000035a3c9 00007fe4a2b08310 00007fe4a2bceed4 00007fe4a2bcef41 00007fe4a2b91f60 00007fe4a2b7d1a9 00007fe4a2b7cc52 00007fe49c202b44
0905 19:59:33.792382 (+ 207us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:33.792623 (+ 241us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:33.792664 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:33.792719 (+ 55us) raft_consensus.cc:1643] Marking committed up to 4
0905 19:59:33.792762 (+ 43us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:33.792832 (+ 70us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:33.792966 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":50041088}
I20250905 19:59:34.143702 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:34.141458 29644 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:34.141170 29678 raft_consensus.cc:491] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:34.144614 28154 raft_consensus.cc:2953] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:34.146623 29575 raft_consensus.cc:2802] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250905 19:59:34.148765 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 91941.2us
I20250905 19:59:34.093309 29617 leader_election.cc:290] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:34.147678 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:34.219120 28152 raft_consensus.cc:2466] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:59:34.150185 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:34.150638 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:34.034890 29685 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:34.104506 29622 raft_consensus.cc:513] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:34.162474 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:34.166015 29688 leader_election.cc:290] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:34.139173 29511 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:34.162101 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:34.138032 29658 raft_consensus.cc:491] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:34.216457 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:34.151201 29514 raft_consensus.cc:513] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:34.170401 28010 raft_consensus.cc:2953] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
W20250905 19:59:34.220242 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 89136.7us
W20250905 19:59:34.221351 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 53209.3us
I20250905 19:59:34.220311 28009 raft_consensus.cc:2466] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 19:59:34.184893 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:34.254117 28156 raft_consensus.cc:2466] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:59:34.167793 29596 raft_consensus.cc:491] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:34.224539 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:34.260643 28003 raft_consensus.cc:2466] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:59:34.260735 27973 leader_election.cc:304] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:34.159574 29629 raft_consensus.cc:2802] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:34.226867 28122 leader_election.cc:304] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:34.146929 29677 leader_election.cc:290] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:34.226787 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:34.234742 29562 leader_election.cc:290] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 19:59:34.250229 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 72491.1us
I20250905 19:59:34.169499 29676 leader_election.cc:290] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:34.141117 27907 rpcz_store.cc:276] 0905 19:59:31.561181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:31.561890 (+ 709us) service_pool.cc:225] Handling call
0905 19:59:34.135115 (+2573225us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20250905 19:59:34.308552 29693 raft_consensus.cc:491] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:34.353806 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:35.580595 28155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 1786) took 1228 ms. Trace:
I20250905 19:59:35.581007 28155 rpcz_store.cc:276] 0905 19:59:34.352046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:34.353519 (+ 1473us) service_pool.cc:225] Handling call
0905 19:59:35.580492 (+1226973us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:34.354804 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:34.245276 29501 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:34.324054 28125 leader_election.cc:304] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 19:59:34.276255 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 49549.3us
I20250905 19:59:34.279562 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:35.783326 28122 leader_election.cc:304] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:35.783768 28151 raft_consensus.cc:2466] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:34.334118 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:59:35.784258 28151 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 1782) took 1507 ms (client timeout 1613 ms). Trace:
W20250905 19:59:35.785171 28151 rpcz_store.cc:269] 0905 19:59:34.276956 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:34.279164 (+ 2208us) service_pool.cc:225] Handling call
0905 19:59:35.784197 (+1505033us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:34.266973 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:34.312897 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:59:35.787635 28159 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 1781) took 1534 ms (client timeout 1812 ms). Trace:
W20250905 19:59:35.788175 28158 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 1783) took 1492 ms (client timeout 1592 ms). Trace:
W20250905 19:59:35.788612 28158 rpcz_store.cc:269] 0905 19:59:34.311428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:34.312593 (+ 1165us) service_pool.cc:225] Handling call
0905 19:59:35.788110 (+1475517us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:35.789162 27904 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:60818 (request call id 144) took 5023 ms. Trace:
W20250905 19:59:34.341463 28745 consensus_meta.cc:287] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.831s user 0.008s sys 0.002s
I20250905 19:59:34.161600 29505 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:34.266261 29450 raft_consensus.cc:2747] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:35.800036 29505 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:34.285528 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:35.801199 28156 raft_consensus.cc:2466] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
W20250905 19:59:35.801862 28752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40904 (request call id 437) took 1517 ms (client timeout 1834 ms). Trace:
W20250905 19:59:35.802254 28752 rpcz_store.cc:269] 0905 19:59:34.284895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:34.285192 (+ 297us) service_pool.cc:225] Handling call
0905 19:59:35.801764 (+1516572us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:34.350906 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:34.260525 29658 raft_consensus.cc:513] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:34.304546 29623 leader_election.cc:290] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:34.271159 29681 leader_election.cc:290] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:34.288368 29680 leader_election.cc:290] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:34.323252 29618 leader_election.cc:290] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:34.291627 29514 leader_election.cc:290] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:34.304796 29666 leader_election.cc:290] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:34.347179 29645 leader_election.cc:290] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:34.356552 29513 leader_election.cc:290] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:34.225328 29678 raft_consensus.cc:513] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:35.816674 28152 raft_consensus.cc:2466] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:59:35.817672 29678 leader_election.cc:290] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 19:59:35.817878 28152 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 1784) took 1489 ms (client timeout 1828 ms). Trace:
I20250905 19:59:34.261096 29511 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:34.167951 29605 raft_consensus.cc:513] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:35.819873 28745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40926 (request call id 305) took 2804 ms. Trace:
I20250905 19:59:35.820798 29511 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:35.821138 29605 leader_election.cc:290] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:34.225091 29644 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:34.262706 29568 leader_election.cc:290] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:35.823868 29568 raft_consensus.cc:491] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:35.824385 29568 raft_consensus.cc:513] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:35.826340 29568 leader_election.cc:290] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:34.243317 29575 raft_consensus.cc:491] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:35.828385 29575 raft_consensus.cc:3058] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:35.829758 27908 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34448 (request call id 251) took 4625 ms. Trace:
I20250905 19:59:34.264983 29596 raft_consensus.cc:513] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:34.264351 29629 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:35.843948 29629 raft_consensus.cc:3058] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:35.844820 29596 leader_election.cc:290] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:34.355645 29624 leader_election.cc:290] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:34.346340 29599 leader_election.cc:290] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:34.178050 29691 raft_consensus.cc:513] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:35.393292 27939 debug-util.cc:398] Leaking SignalData structure 0x7b08002d0060 after lost signal to thread 26858
W20250905 19:59:35.849692 27939 debug-util.cc:398] Leaking SignalData structure 0x7b08002fa340 after lost signal to thread 26861
W20250905 19:59:35.398582 28010 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.067s user 0.013s sys 0.004s
W20250905 19:59:35.850782 28010 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.519s user 0.014s sys 0.004s
I20250905 19:59:35.851027 29691 leader_election.cc:290] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:35.851608 28010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1151) took 1718 ms. Trace:
I20250905 19:59:35.851845 28747 raft_consensus.cc:1273] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250905 19:59:35.852005 28010 rpcz_store.cc:276] 0905 19:59:34.133384 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:34.169742 (+ 36358us) service_pool.cc:225] Handling call
0905 19:59:34.170079 (+ 337us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:34.170283 (+ 204us) raft_consensus.cc:1532] Early marking committed up to index 3
0905 19:59:34.331577 (+161294us) log.cc:844] Serialized 14 byte log entry
0905 19:59:35.851258 (+1519681us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:35.851313 (+ 55us) raft_consensus.cc:1643] Marking committed up to 3
0905 19:59:35.851359 (+ 46us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:35.851438 (+ 79us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:35.851553 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1519255,"threads_started":1,"wal-append.queue_time_us":74703,"wal-append.run_cpu_time_us":1095,"wal-append.run_wall_time_us":1001154}
I20250905 19:59:35.433532 28001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 137) took 1619 ms. Trace:
I20250905 19:59:35.853508 28001 rpcz_store.cc:276] 0905 19:59:33.814444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:34.063216 (+248772us) service_pool.cc:225] Handling call
0905 19:59:34.063607 (+ 391us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:34.127386 (+ 63779us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:34.127433 (+ 47us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:34.127496 (+ 63us) raft_consensus.cc:1643] Marking committed up to 6
0905 19:59:34.127537 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:35.433163 (+1305626us) spinlock_profiling.cc:243] Waited 63.5 ms on lock 0x7b540016e9d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:35.433224 (+ 61us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:35.433417 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":146083200}
I20250905 19:59:35.437862 28153 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 1771) took 1294 ms. Trace:
I20250905 19:59:35.854679 28153 rpcz_store.cc:276] 0905 19:59:34.143734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:34.143851 (+ 117us) service_pool.cc:225] Handling call
0905 19:59:34.144123 (+ 272us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:34.164628 (+ 20505us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:34.164659 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:34.164700 (+ 41us) raft_consensus.cc:1643] Marking committed up to 1
0905 19:59:34.164729 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:35.437521 (+1272792us) spinlock_profiling.cc:243] Waited 20.3 ms on lock 0x7b5400326c54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:35.437582 (+ 61us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:35.437780 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":46640896}
I20250905 19:59:35.454197 29685 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:35.464169 28749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40904 (request call id 429) took 1536 ms. Trace:
I20250905 19:59:35.510469 29693 raft_consensus.cc:513] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:35.856374 28749 rpcz_store.cc:276] 0905 19:59:33.927813 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:33.928094 (+ 281us) service_pool.cc:225] Handling call
0905 19:59:33.928486 (+ 392us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:34.114872 (+186386us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:34.114901 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:34.114941 (+ 40us) raft_consensus.cc:1643] Marking committed up to 4
0905 19:59:34.114973 (+ 32us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:35.463865 (+1348892us) spinlock_profiling.cc:243] Waited 186 ms on lock 0x7b54004e07d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:35.463921 (+ 56us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:35.464095 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":428063488}
I20250905 19:59:35.857959 29693 leader_election.cc:290] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:35.546967 28004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1132) took 1989 ms. Trace:
I20250905 19:59:35.858708 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:35.858592 28004 rpcz_store.cc:276] 0905 19:59:33.557539 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:33.557661 (+ 122us) service_pool.cc:225] Handling call
0905 19:59:33.557930 (+ 269us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:34.138610 (+580680us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:34.138640 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:34.138686 (+ 46us) raft_consensus.cc:1643] Marking committed up to 2
0905 19:59:34.138716 (+ 30us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:35.546619 (+1407903us) spinlock_profiling.cc:243] Waited 580 ms on lock 0x7b5400291654. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:35.546687 (+ 68us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:35.546889 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1334844288}
W20250905 19:59:35.859547 27939 debug-util.cc:398] Leaking SignalData structure 0x7b080031cd80 after lost signal to thread 27942
I20250905 19:59:35.582896 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:59:35.585351 28123 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.361s user 0.006s sys 0.015s
W20250905 19:59:35.859997 27939 debug-util.cc:398] Leaking SignalData structure 0x7b08002ee9a0 after lost signal to thread 27956
W20250905 19:59:35.860244 28123 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.636s user 0.007s sys 0.016s
W20250905 19:59:35.860329 28155 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33752 (request call id 1292) took 1506 ms (client timeout 1536 ms). Trace:
W20250905 19:59:35.860777 28155 rpcz_store.cc:269] 0905 19:59:34.353817 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:35.582554 (+1228737us) service_pool.cc:225] Handling call
0905 19:59:35.860267 (+277713us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:35.861177 27939 debug-util.cc:398] Leaking SignalData structure 0x7b0800206d00 after lost signal to thread 28034
W20250905 19:59:35.861505 27939 debug-util.cc:398] Leaking SignalData structure 0x7b0800206d40 after lost signal to thread 28035
I20250905 19:59:35.861814 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:59:34.386787 29619 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.279s user 0.010s sys 0.043s
W20250905 19:59:35.862860 29619 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 2.761s user 0.011s sys 0.052s
I20250905 19:59:35.594960 29690 raft_consensus.cc:491] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:35.863937 29690 raft_consensus.cc:513] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:35.865043 28155 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 1790) took 1511 ms (client timeout 1908 ms). Trace:
W20250905 19:59:35.865495 28155 rpcz_store.cc:269] 0905 19:59:34.353204 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:35.861514 (+1508310us) service_pool.cc:225] Handling call
0905 19:59:35.864971 (+ 3457us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:35.604945 28154 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.250s user 0.015s sys 0.011s
W20250905 19:59:35.866982 28154 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.512s user 0.016s sys 0.011s
I20250905 19:59:35.867388 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:35.616348 29710 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
W20250905 19:59:35.620558 28003 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1777) took 1755 ms (client timeout 1618 ms). Trace:
I20250905 19:59:35.867990 29566 raft_consensus.cc:2747] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250905 19:59:35.868157 28003 rpcz_store.cc:269] 0905 19:59:33.865171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:34.004849 (+139678us) service_pool.cc:225] Handling call
0905 19:59:35.620283 (+1615434us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b54001fc054. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:35.620437 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":577910656}
I20250905 19:59:35.868204 29710 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:35.627775 29635 leader_election.cc:290] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:35.639243 29372 raft_consensus.cc:491] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:35.842914 28050 leader_election.cc:304] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:35.870849 29372 raft_consensus.cc:513] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:35.871341 29513 raft_consensus.cc:2802] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:35.871834 29513 raft_consensus.cc:491] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:35.871943 28050 leader_election.cc:304] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:35.872273 29513 raft_consensus.cc:3058] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:35.873322 29372 leader_election.cc:290] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:35.653350 29622 leader_election.cc:290] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:35.664217 29713 raft_consensus.cc:491] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:35.666891 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:35.874372 29713 raft_consensus.cc:513] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:35.874063 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:35.670219 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:35.875844 28750 raft_consensus.cc:2466] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 19:59:35.669275 29714 raft_consensus.cc:491] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:35.877202 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:35.877007 29714 raft_consensus.cc:513] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:35.674343 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:35.879244 29714 leader_election.cc:290] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:35.683277 29715 raft_consensus.cc:491] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:35.675452 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:35.695449 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:35.879984 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:35.879848 29715 raft_consensus.cc:513] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:35.880578 28083 raft_consensus.cc:2466] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:35.873375 29511 raft_consensus.cc:2802] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:35.881672 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:35.882081 29715 leader_election.cc:290] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:35.695221 28007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1141) took 2031 ms. Trace:
I20250905 19:59:35.697688 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:35.882905 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:35.883535 28083 raft_consensus.cc:2466] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 19:59:35.884377 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:35.884929 28083 raft_consensus.cc:2466] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:59:35.885699 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:35.697688 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:35.703661 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:35.880654 28155 raft_consensus.cc:2466] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:59:35.887292 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:35.887606 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:35.887943 28076 raft_consensus.cc:2466] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:35.888803 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:35.888949 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:35.889148 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:35.889899 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:35.698447 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:35.876951 29713 leader_election.cc:290] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:35.890632 28082 raft_consensus.cc:2466] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:35.891530 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:35.892386 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:35.892798 28082 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 19:59:35.893302 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:35.893671 28082 raft_consensus.cc:2466] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
W20250905 19:59:35.699414 27975 connection.cc:232] RPC call timeout handler was delayed by -1.1985111240001061s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15360000000000001s for a response
I20250905 19:59:35.722306 29717 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:35.894848 29717 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:35.723444 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:35.707136 29501 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:35.721874 29512 leader_election.cc:290] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:35.906567 29618 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:35.727890 29711 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:35.728470 29508 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:35.908243 29618 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:35.910264 29508 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:35.728571 29609 raft_consensus.cc:491] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:35.911881 29609 raft_consensus.cc:513] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:35.742856 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:35.912909 28081 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:35.913070 28002 raft_consensus.cc:2466] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
W20250905 19:59:35.913653 28002 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1149) took 1793 ms (client timeout 2245 ms). Trace:
I20250905 19:59:35.744091 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:35.738525 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:35.745404 29688 raft_consensus.cc:491] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:35.914392 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:35.914785 29688 raft_consensus.cc:513] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:35.915161 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:35.914227 29609 leader_election.cc:290] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:35.915593 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:35.916213 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:35.744218 29718 raft_consensus.cc:491] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:35.916836 29688 leader_election.cc:290] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:35.761912 29627 raft_consensus.cc:491] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:35.917273 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:35.917158 29718 raft_consensus.cc:513] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:35.917574 29627 raft_consensus.cc:513] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:35.917943 28079 raft_consensus.cc:2466] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:59:35.762372 29720 raft_consensus.cc:491] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:35.772620 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:35.777894 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 1.45383e+06us
I20250905 19:59:35.918932 27975 leader_election.cc:304] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:35.919132 28078 raft_consensus.cc:2466] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:59:35.779299 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:35.778734 29481 raft_consensus.cc:2747] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:35.784531 28122 leader_election.cc:304] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:34.312502 28157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1290) took 1252 ms. Trace:
I20250905 19:59:35.789230 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:35.928375 28151 raft_consensus.cc:2466] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
I20250905 19:59:35.928007 29720 raft_consensus.cc:513] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:35.928988 28151 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 1787) took 1576 ms (client timeout 1653 ms). Trace:
I20250905 19:59:35.789530 27904 rpcz_store.cc:276] 0905 19:59:30.765917 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:30.766058 (+ 141us) service_pool.cc:225] Handling call
0905 19:59:35.789099 (+5023041us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":6}
I20250905 19:59:35.789628 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:35.782011 29719 raft_consensus.cc:491] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:35.928079 28157 rpcz_store.cc:276] 0905 19:59:33.058633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:33.987477 (+928844us) service_pool.cc:225] Handling call
0905 19:59:33.987838 (+ 361us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:34.085344 (+ 97506us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:34.085393 (+ 49us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:34.085456 (+ 63us) raft_consensus.cc:1643] Marking committed up to 4
0905 19:59:34.085501 (+ 45us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:34.310769 (+225268us) spinlock_profiling.cc:243] Waited 97.2 ms on lock 0x7b540009d3d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:34.310828 (+ 59us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:34.310969 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":223537536}
W20250905 19:59:35.803251 28156 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 1785) took 1451 ms (client timeout 1884 ms). Trace:
W20250905 19:59:35.931382 28156 rpcz_store.cc:269] 0905 19:59:34.351636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:34.354520 (+ 2884us) service_pool.cc:225] Handling call
0905 19:59:35.803100 (+1448580us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b5400186354. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:35.803199 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":322271616}
W20250905 19:59:35.804872 28744 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40904 (request call id 439) took 1486 ms (client timeout 1658 ms). Trace:
W20250905 19:59:35.933010 28744 rpcz_store.cc:269] 0905 19:59:34.318852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:34.323322 (+ 4470us) service_pool.cc:225] Handling call
0905 19:59:35.804807 (+1481485us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:35.933223 28158 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33752 (request call id 1293) took 1579 ms (client timeout 1894 ms). Trace:
W20250905 19:59:35.806293 27973 leader_election.cc:336] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.595s (SENT)
W20250905 19:59:35.933789 28158 rpcz_store.cc:269] 0905 19:59:34.354077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:35.789359 (+1435282us) service_pool.cc:225] Handling call
0905 19:59:35.933157 (+143798us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:35.934105 27973 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:35.934355 29720 leader_election.cc:290] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:35.806988 29658 leader_election.cc:290] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:35.807668 29505 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 19:59:35.788180 28159 rpcz_store.cc:269] 0905 19:59:34.253405 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:34.253592 (+ 187us) service_pool.cc:225] Handling call
0905 19:59:35.787579 (+1533987us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:35.814579 29514 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:35.936630 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:35.937850 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:59:35.818318 28152 rpcz_store.cc:269] 0905 19:59:34.328614 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:34.328903 (+ 289us) service_pool.cc:225] Handling call
0905 19:59:35.817701 (+1488798us) spinlock_profiling.cc:243] Waited 31.6 ms on lock 0x7b5400326254. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:35.817817 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":72569600}
I20250905 19:59:35.929677 29713 raft_consensus.cc:2747] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:35.939030 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:35.940240 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:35.940786 28152 raft_consensus.cc:2466] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:59:35.941672 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:35.821354 29716 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:34.169436 29590 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:35.820394 28745 rpcz_store.cc:276] 0905 19:59:33.015855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:33.016737 (+ 882us) service_pool.cc:225] Handling call
0905 19:59:33.017707 (+ 970us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:33.018024 (+ 317us) raft_consensus.cc:1532] Early marking committed up to index 3
0905 19:59:35.797081 (+2779057us) log.cc:844] Serialized 14 byte log entry
0905 19:59:35.815993 (+ 18912us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:35.816061 (+ 68us) raft_consensus.cc:1643] Marking committed up to 3
0905 19:59:35.816144 (+ 83us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:35.816399 (+ 255us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:35.819714 (+ 3315us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2721,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":91,"thread_start_us":18483,"threads_started":1,"wal-append.queue_time_us":15127}
I20250905 19:59:35.947732 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:35.936520 29514 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:35.832077 29644 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 19:59:35.246985 28051 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.976s user 0.024s sys 0.041s
I20250905 19:59:35.948930 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:35.951553 29718 leader_election.cc:290] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:35.931103 29678 raft_consensus.cc:2802] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:35.952512 29678 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
W20250905 19:59:35.952898 28051 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.685s user 0.028s sys 0.041s
I20250905 19:59:35.953034 29678 raft_consensus.cc:3058] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:35.842628 27908 rpcz_store.cc:276] 0905 19:59:31.204824 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:31.205034 (+ 210us) service_pool.cc:225] Handling call
0905 19:59:35.829682 (+4624648us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":371593,"reported_tablets":6}
I20250905 19:59:35.853144 28747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40918 (request call id 364) took 1550 ms. Trace:
I20250905 19:59:35.858040 29574 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:35.955113 28747 rpcz_store.cc:276] 0905 19:59:34.303030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:34.304005 (+ 975us) service_pool.cc:225] Handling call
0905 19:59:34.334139 (+ 30134us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:35.852268 (+1518129us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:35.852910 (+ 642us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b540049d4d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:35.853084 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3471200640}
I20250905 19:59:35.943091 29627 leader_election.cc:290] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:35.956357 29574 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:35.957731 29627 consensus_queue.cc:1035] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250905 19:59:35.958830 29450 raft_consensus.cc:2747] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:35.926115 29618 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:35.867661 28154 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 1772) took 1723 ms. Trace:
I20250905 19:59:35.962673 28154 rpcz_store.cc:276] 0905 19:59:34.143933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:34.144128 (+ 195us) service_pool.cc:225] Handling call
0905 19:59:34.144336 (+ 208us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:34.144515 (+ 179us) raft_consensus.cc:1532] Early marking committed up to index 3
0905 19:59:34.354492 (+209977us) log.cc:844] Serialized 14 byte log entry
0905 19:59:35.867416 (+1512924us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:35.867461 (+ 45us) raft_consensus.cc:1643] Marking committed up to 3
0905 19:59:35.867487 (+ 26us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:35.867543 (+ 56us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:35.867618 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2224,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":110,"thread_start_us":1512535,"threads_started":1,"wal-append.queue_time_us":51507,"wal-append.run_cpu_time_us":1289,"wal-append.run_wall_time_us":1001300}
I20250905 19:59:35.965341 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:35.612504 28085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1202) took 2189 ms. Trace:
I20250905 19:59:35.966634 28085 rpcz_store.cc:276] 0905 19:59:33.423034 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:33.423251 (+ 217us) service_pool.cc:225] Handling call
0905 19:59:34.128253 (+705002us) spinlock_profiling.cc:243] Waited 2.55 ms on lock 0x7b5800204f98. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b91f60 00007fe4a2b7d1a9 00007fe4a2b7cc52 00007fe49c202b44
0905 19:59:34.128535 (+ 282us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:34.257952 (+129417us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:34.257987 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:34.258040 (+ 53us) raft_consensus.cc:1643] Marking committed up to 2
0905 19:59:34.258071 (+ 31us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:35.612160 (+1354089us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b540023c154. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:35.612217 (+ 57us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:35.612396 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":302903936}
I20250905 19:59:35.871207 29710 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:35.667357 28150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:33742 (request call id 618) took 2682 ms. Trace:
I20250905 19:59:35.875093 28050 leader_election.cc:304] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 19:59:35.969010 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 93972.6us
I20250905 19:59:35.675472 27905 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34440 (request call id 249) took 4539 ms. Trace:
W20250905 19:59:35.969344 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 1.7038e+06us
I20250905 19:59:35.969636 27905 rpcz_store.cc:276] 0905 19:59:31.135872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:31.147612 (+ 11740us) service_pool.cc:225] Handling call
0905 19:59:35.675379 (+4527767us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1468098,"reported_tablets":1}
I20250905 19:59:35.881799 29511 raft_consensus.cc:491] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:35.968693 28150 rpcz_store.cc:276] 0905 19:59:32.985385 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:33.911480 (+926095us) service_pool.cc:225] Handling call
0905 19:59:35.666744 (+1755264us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b5400316354. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9dcd0 00007fe49dc21eea 00007fe4a2b8161e 00007fe49c217cc7 00007fe49c217c52 00007fe49c217b81 00007fe49c217afc 00007fe49c216db2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:35.667178 (+ 434us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":476691328}
I20250905 19:59:35.971195 29511 raft_consensus.cc:3058] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:35.971244 29627 raft_consensus.cc:2802] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:35.898108 28748 raft_consensus.cc:2466] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 19:59:35.972453 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:59:35.972735 28748 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40926 (request call id 311) took 2238 ms (client timeout 2185 ms). Trace:
I20250905 19:59:35.726365 28077 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1201) took 2493 ms. Trace:
I20250905 19:59:35.974495 28050 leader_election.cc:304] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:35.727013 29626 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:35.976298 28050 leader_election.cc:304] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:35.882674 28007 rpcz_store.cc:276] 0905 19:59:33.664287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:33.741519 (+ 77232us) service_pool.cc:225] Handling call
0905 19:59:34.050853 (+309334us) spinlock_profiling.cc:243] Waited 37.2 ms on lock 0x7b5800000f98. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b91f60 00007fe4a2b7d1a9 00007fe4a2b7cc52 00007fe49c202b44
0905 19:59:35.623064 (+1572211us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:35.623265 (+ 201us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:35.623294 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:35.623342 (+ 48us) raft_consensus.cc:1643] Marking committed up to 1
0905 19:59:35.623371 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:35.623435 (+ 64us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:35.695049 (+ 71614us) spinlock_profiling.cc:243] Waited 1.57 s on lock 0x7b54001fc050. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:35.695160 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3700768384}
I20250905 19:59:35.894327 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:35.902125 27939 debug-util.cc:398] Leaking SignalData structure 0x7b08001c4e20 after lost signal to thread 28109
I20250905 19:59:35.902597 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:35.904083 29501 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:35.904050 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:35.905639 29685 raft_consensus.cc:491] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:35.906077 29680 raft_consensus.cc:491] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:35.906535 29681 raft_consensus.cc:491] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:35.983419 29680 raft_consensus.cc:513] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:35.984277 29681 raft_consensus.cc:513] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:35.984930 29685 raft_consensus.cc:513] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:35.985782 29680 leader_election.cc:290] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:35.905400 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:35.987005 29685 leader_election.cc:290] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:35.981961 29626 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:35.987412 28153 raft_consensus.cc:2466] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:59:35.987836 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:35.988343 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:35.866194 29690 leader_election.cc:290] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:35.988907 28080 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 19:59:35.989372 29626 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:35.890161 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:35.981958 29718 raft_consensus.cc:2802] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:35.908016 29717 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:35.912235 29508 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:35.912415 29711 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:35.990521 29718 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:35.991144 29718 raft_consensus.cc:3058] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:35.749629 28718 leader_election.cc:304] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:35.919946 28084 raft_consensus.cc:2466] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:59:35.784198 29559 raft_consensus.cc:2802] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:35.996897 29574 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:35.997315 29559 raft_consensus.cc:491] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:35.997813 29559 raft_consensus.cc:3058] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:35.916852 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:35.913996 28002 rpcz_store.cc:269] 0905 19:59:34.120140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:34.160835 (+ 40695us) service_pool.cc:225] Handling call
0905 19:59:35.742666 (+1581831us) spinlock_profiling.cc:243] Waited 1.12 ms on lock 0x7b5800000f98. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b7db60 00007fe4a2b7d444 00007fe49c2050c4 00007fe49c205052
0905 19:59:35.913598 (+170932us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2574592}
I20250905 19:59:35.914153 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 19:59:35.924321 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 139857us
W20250905 19:59:35.929374 28151 rpcz_store.cc:269] 0905 19:59:34.352374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:35.788904 (+1436530us) service_pool.cc:225] Handling call
0905 19:59:35.928896 (+139992us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:35.925550 28125 leader_election.cc:304] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:35.935031 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:35.935165 29719 raft_consensus.cc:513] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:35.943675 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 128971us
W20250905 19:59:36.001708 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 76273.5us
I20250905 19:59:36.001950 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.002612 28151 raft_consensus.cc:2466] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:59:36.003455 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.004009 28151 raft_consensus.cc:2466] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:59:35.944201 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:35.946002 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:35.942726 29716 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:35.946985 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:35.947324 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:35.949597 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:35.949738 29514 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:35.951200 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:35.952008 28159 raft_consensus.cc:2466] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:59:35.857955 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:35.972054 29627 raft_consensus.cc:491] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:35.966238 28154 raft_consensus.cc:2466] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
I20250905 19:59:36.011396 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:35.998296 29717 raft_consensus.cc:2802] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:36.019556 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.019886 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.020253 28151 raft_consensus.cc:2466] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:35.973135 28150 raft_consensus.cc:2466] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
W20250905 19:59:35.973117 28748 rpcz_store.cc:269] 0905 19:59:33.734789 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:35.723093 (+1988304us) service_pool.cc:225] Handling call
0905 19:59:35.972668 (+249575us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:35.841176 29575 raft_consensus.cc:513] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:35.979595 28050 leader_election.cc:304] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 19:59:36.024667 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 45274.1us
W20250905 19:59:35.979568 27939 debug-util.cc:398] Leaking SignalData structure 0x7b0800324cc0 after lost signal to thread 28183
I20250905 19:59:35.977692 27975 leader_election.cc:304] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:35.981056 28155 raft_consensus.cc:2466] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 19:59:35.974445 28077 rpcz_store.cc:276] 0905 19:59:33.232793 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:33.235238 (+ 2445us) service_pool.cc:225] Handling call
0905 19:59:34.140588 (+905350us) spinlock_profiling.cc:243] Waited 4.39 ms on lock 0x7b5800204f98. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac6721e 000000000035a3c9 00007fe4a2b08310 00007fe4a2bceed4 00007fe4a2bcef41 00007fe4a2b91f60 00007fe4a2b7d1a9 00007fe4a2b7cc52 00007fe49c202b44
0905 19:59:34.149885 (+ 9297us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:34.271277 (+121392us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:34.276260 (+ 4983us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:34.280896 (+ 4636us) raft_consensus.cc:1643] Marking committed up to 1
0905 19:59:34.287891 (+ 6995us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:35.725983 (+1438092us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b540029a754. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:35.726054 (+ 71us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:35.726197 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":251994368}
W20250905 19:59:35.992288 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 242778us
I20250905 19:59:35.988919 28153 raft_consensus.cc:2466] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:59:36.027546 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:36.024085 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.028218 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.028302 28002 raft_consensus.cc:2466] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 1.
I20250905 19:59:36.029485 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:35.994704 28083 raft_consensus.cc:2408] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 2 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20250905 19:59:35.995132 29711 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:36.077531 28152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1294) took 1723 ms. Trace:
I20250905 19:59:36.084666 29629 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:36.084883 28152 rpcz_store.cc:276] 0905 19:59:34.354372 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:36.044883 (+1690511us) service_pool.cc:225] Handling call
0905 19:59:36.077067 (+ 32184us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:36.077236 (+ 169us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:36.077264 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:36.077303 (+ 39us) raft_consensus.cc:1643] Marking committed up to 2
0905 19:59:36.077330 (+ 27us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:36.077388 (+ 58us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:36.077479 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:35.996695 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:35.999373 28122 leader_election.cc:304] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:36.000703 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:36.086788 28743 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
W20250905 19:59:36.087378 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 88096.8us
I20250905 19:59:36.086794 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:36.088312 28009 raft_consensus.cc:2466] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 1.
I20250905 19:59:36.039620 29514 raft_consensus.cc:2802] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:36.089581 29514 raft_consensus.cc:491] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:36.089979 29514 raft_consensus.cc:3058] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:36.090716 29501 raft_consensus.cc:2802] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:36.091145 29501 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:36.091514 29501 raft_consensus.cc:3058] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:36.008920 28125 leader_election.cc:304] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:35.981114 29507 raft_consensus.cc:2747] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:35.970165 28051 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 19:59:36.025676 27939 debug-util.cc:398] Leaking SignalData structure 0x7b080027c260 after lost signal to thread 28184
I20250905 19:59:36.010325 29658 raft_consensus.cc:2747] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250905 19:59:36.092924 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 122853us
W20250905 19:59:36.093528 27939 debug-util.cc:398] Leaking SignalData structure 0x7b08001ce160 after lost signal to thread 28776
W20250905 19:59:36.099457 27939 debug-util.cc:398] Leaking SignalData structure 0x7b08001310a0 after lost signal to thread 29671
I20250905 19:59:36.070925 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:36.033044 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:36.038985 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.073225 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.073995 28154 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1296) took 1709 ms. Trace:
I20250905 19:59:36.069914 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.109278 28150 raft_consensus.cc:2466] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:59:35.984086 29513 raft_consensus.cc:513] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:36.116149 29513 leader_election.cc:290] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:36.116600 29678 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:36.109022 28154 rpcz_store.cc:276] 0905 19:59:34.364892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:36.059920 (+1695028us) service_pool.cc:225] Handling call
0905 19:59:36.073351 (+ 13431us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:36.073544 (+ 193us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:36.073569 (+ 25us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:36.073612 (+ 43us) raft_consensus.cc:1643] Marking committed up to 1
0905 19:59:36.073661 (+ 49us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:36.073743 (+ 82us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:36.073918 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3840}
I20250905 19:59:36.076133 28157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1295) took 1714 ms. Trace:
I20250905 19:59:36.076359 29716 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:36.078341 28153 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1291) took 1724 ms. Trace:
I20250905 19:59:36.035050 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:59:36.078537 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 100945us
I20250905 19:59:36.082334 28123 leader_election.cc:304] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 19:59:36.082579 29574 raft_consensus.cc:2802] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:36.083405 28158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 1788) took 1730 ms. Trace:
W20250905 19:59:36.135242 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 53001.2us
W20250905 19:59:36.123934 28715 leader_election.cc:336] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.811s (SENT)
I20250905 19:59:36.019420 29627 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:36.136269 29678 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:36.136912 29629 leader_election.cc:290] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:36.106912 28005 raft_consensus.cc:2466] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 1.
W20250905 19:59:36.092494 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 83684.7us
I20250905 19:59:36.099156 28051 leader_election.cc:304] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:36.011528 29599 consensus_queue.cc:1035] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.081s
I20250905 19:59:36.096271 29711 raft_consensus.cc:2747] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:36.138693 28158 rpcz_store.cc:276] 0905 19:59:34.352657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:36.069501 (+1716844us) service_pool.cc:225] Handling call
0905 19:59:36.082870 (+ 13369us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:36.083056 (+ 186us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:36.083099 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:36.083144 (+ 45us) raft_consensus.cc:1643] Marking committed up to 1
0905 19:59:36.083181 (+ 37us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:36.083239 (+ 58us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:36.083344 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:36.102555 29719 leader_election.cc:290] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:36.144078 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:36.144937 29507 raft_consensus.cc:2747] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:36.146216 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:36.146942 28008 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 1.
I20250905 19:59:36.147637 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:36.019661 29717 raft_consensus.cc:491] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:36.104094 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:36.148815 29717 raft_consensus.cc:3058] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:36.149641 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:36.150288 28005 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 2.
I20250905 19:59:36.075656 28155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 1789) took 1722 ms. Trace:
I20250905 19:59:36.151042 28155 rpcz_store.cc:276] 0905 19:59:34.352927 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:36.042335 (+1689408us) service_pool.cc:225] Handling call
0905 19:59:36.075133 (+ 32798us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:36.075340 (+ 207us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:36.075364 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:36.075404 (+ 40us) raft_consensus.cc:1643] Marking committed up to 1
0905 19:59:36.075433 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:36.075494 (+ 61us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:36.075599 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:36.118678 28079 raft_consensus.cc:1238] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:36.030606 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:36.152726 28007 raft_consensus.cc:2466] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 2.
I20250905 19:59:36.135361 29505 raft_consensus.cc:2802] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:36.153491 29505 raft_consensus.cc:491] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:36.153923 29505 raft_consensus.cc:3058] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:36.136224 29574 raft_consensus.cc:491] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:36.099622 29512 raft_consensus.cc:2802] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:36.160593 29512 raft_consensus.cc:491] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:36.161074 29512 raft_consensus.cc:3058] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
W20250905 19:59:36.082005 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 56492.6us
I20250905 19:59:36.122270 28153 rpcz_store.cc:276] 0905 19:59:34.353542 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:36.039804 (+1686262us) service_pool.cc:225] Handling call
0905 19:59:36.076259 (+ 36455us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:36.078049 (+ 1790us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:36.078076 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:36.078113 (+ 37us) raft_consensus.cc:1643] Marking committed up to 4
0905 19:59:36.078143 (+ 30us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:36.078193 (+ 50us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:36.078291 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:36.138998 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:36.139179 28051 leader_election.cc:304] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:36.162942 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.164976 28050 leader_election.cc:304] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:36.167302 29508 raft_consensus.cc:2747] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:36.101693 29681 leader_election.cc:290] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:36.101243 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:36.163161 29574 raft_consensus.cc:3058] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:36.140585 28715 leader_election.cc:304] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:36.137382 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:36.148244 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:36.084681 29575 leader_election.cc:290] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:36.155660 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.181808 28155 raft_consensus.cc:2466] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
W20250905 19:59:36.156215 28122 connection.cc:232] RPC call timeout handler was delayed by -1.1239441280000619s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18530000000000002s for a response
W20250905 19:59:36.182682 28122 connection.cc:232] RPC call timeout handler was delayed by -1.0212118790000204s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18460000000000001s for a response
I20250905 19:59:36.159678 28718 leader_election.cc:304] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:36.166644 28051 leader_election.cc:304] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:36.166762 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:36.122012 28157 rpcz_store.cc:276] 0905 19:59:34.361712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:36.038690 (+1676978us) service_pool.cc:225] Handling call
0905 19:59:36.075687 (+ 36997us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:36.075851 (+ 164us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:36.075874 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:36.075918 (+ 44us) raft_consensus.cc:1643] Marking committed up to 2
0905 19:59:36.075946 (+ 28us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:36.075993 (+ 47us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:36.076080 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:36.140029 28125 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:36.167640 29710 raft_consensus.cc:2802] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:36.168253 28050 leader_election.cc:304] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:36.030887 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 19:59:36.169250 29729 log.cc:927] Time spent T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20250905 19:59:36.187247 29710 raft_consensus.cc:491] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:36.163473 29501 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:36.196333 29627 raft_consensus.cc:513] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:36.197660 29501 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:36.173687 29511 raft_consensus.cc:513] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:36.178117 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 54223.7us
I20250905 19:59:36.180647 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.181067 28008 raft_consensus.cc:2391] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 2: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:36.185774 28051 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 19:59:36.186096 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 46162.7us
I20250905 19:59:36.203258 27973 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:36.189699 29716 raft_consensus.cc:2802] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:36.187764 29710 raft_consensus.cc:3058] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:36.188859 29693 raft_consensus.cc:2747] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:36.190837 29626 raft_consensus.cc:2747] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:36.186584 29688 raft_consensus.cc:2747] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:36.206761 27973 leader_election.cc:304] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:36.191839 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.192867 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.207159 28051 leader_election.cc:304] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:36.193879 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.194689 29644 raft_consensus.cc:2802] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:36.194973 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.199373 28718 leader_election.cc:304] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:36.209573 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.209832 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.210760 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.219923 28155 raft_consensus.cc:2391] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:36.220194 28156 raft_consensus.cc:2466] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:36.220482 28125 leader_election.cc:304] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:36.211653 28153 raft_consensus.cc:2391] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 4: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:36.209399 27975 leader_election.cc:304] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:36.212540 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.213724 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.209164 27905 catalog_manager.cc:5582] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:36.214918 29644 raft_consensus.cc:491] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:36.223783 29568 raft_consensus.cc:2802] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:36.223958 29644 raft_consensus.cc:3058] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:36.224278 29568 raft_consensus.cc:491] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:36.215389 27973 leader_election.cc:304] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:36.216544 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:36.217020 29681 raft_consensus.cc:2802] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:36.207540 28157 raft_consensus.cc:2466] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
W20250905 19:59:36.262938 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 53624.3us
I20250905 19:59:36.245821 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.228060 27905 catalog_manager.cc:5582] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:36.264132 27975 leader_election.cc:304] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:36.168025 29514 raft_consensus.cc:513] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:36.264918 28078 raft_consensus.cc:1238] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:36.271224 28157 raft_consensus.cc:1238] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:36.271476 27975 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:36.198156 29559 raft_consensus.cc:513] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:36.264948 28154 raft_consensus.cc:2466] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 1.
I20250905 19:59:36.227686 28718 leader_election.cc:304] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:36.276918 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:59:36.280035 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 64703.1us
I20250905 19:59:36.280889 27973 leader_election.cc:304] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:36.280745 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.278537 28151 raft_consensus.cc:2466] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 1.
I20250905 19:59:36.282948 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.211143 29575 raft_consensus.cc:2802] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250905 19:59:36.284397 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 56851.8us
I20250905 19:59:36.204530 29716 raft_consensus.cc:491] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:36.262413 29568 raft_consensus.cc:3058] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:36.300027 28718 leader_election.cc:304] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:36.218770 29507 raft_consensus.cc:2747] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 19:59:36.301744 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.303210 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.262699 29681 raft_consensus.cc:491] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:36.304792 27975 leader_election.cc:304] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:36.309504 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.306613 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.273149 29629 raft_consensus.cc:2802] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:36.273940 29688 raft_consensus.cc:2802] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:36.309653 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.325620 28084 raft_consensus.cc:2466] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:59:36.326917 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.310703 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.335196 28080 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 19:59:36.335309 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.275869 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.336236 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:36.337318 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.337421 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.337987 28083 raft_consensus.cc:2466] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:59:36.338402 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.342893 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.343375 28080 raft_consensus.cc:2466] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:36.284328 29604 raft_consensus.cc:2747] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250905 19:59:36.313553 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 106454us
I20250905 19:59:36.300025 29590 raft_consensus.cc:2802] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:36.349020 28051 leader_election.cc:304] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:36.276672 29668 raft_consensus.cc:2802] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:36.318569 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.316589 29681 raft_consensus.cc:3058] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:36.312791 28152 raft_consensus.cc:1238] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:36.300117 29690 raft_consensus.cc:2802] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:36.361827 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.364223 29690 raft_consensus.cc:491] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:36.364748 29690 raft_consensus.cc:3058] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:36.300307 29685 raft_consensus.cc:2802] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:36.325286 29688 raft_consensus.cc:491] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:36.352212 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.372500 28081 raft_consensus.cc:2466] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:59:36.352627 28078 raft_consensus.cc:1238] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:36.352941 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.374374 28080 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:36.375651 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.377346 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.305897 29575 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:36.355090 28085 raft_consensus.cc:2466] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:36.379518 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.379671 28050 leader_election.cc:304] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:36.323652 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.380913 28082 raft_consensus.cc:2466] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:36.314417 29714 raft_consensus.cc:2802] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:36.357000 29668 raft_consensus.cc:491] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:36.386698 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.275465 29718 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:36.357160 29720 raft_consensus.cc:2747] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:36.360980 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.361814 28051 leader_election.cc:304] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:36.342936 29626 raft_consensus.cc:2747] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:36.313627 29629 raft_consensus.cc:491] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:36.301230 29716 raft_consensus.cc:3058] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:36.355500 28076 raft_consensus.cc:2466] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 19:59:36.352063 29590 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:36.391568 28051 leader_election.cc:304] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:36.406622 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.408560 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.392234 28050 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:36.367621 29685 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:36.415182 28050 leader_election.cc:304] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:36.403079 29718 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:36.368688 29604 raft_consensus.cc:2802] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:36.409269 28085 raft_consensus.cc:2466] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:36.382570 29688 raft_consensus.cc:3058] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:36.400413 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.411334 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:36.403652 29507 raft_consensus.cc:2747] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:36.414705 28084 raft_consensus.cc:2466] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:36.399277 29627 leader_election.cc:290] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:36.415942 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.385677 29575 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:36.378122 28080 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:59:36.390362 29508 raft_consensus.cc:2802] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:36.386999 29714 raft_consensus.cc:491] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:36.391302 29668 raft_consensus.cc:3058] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:36.425663 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:36.420050 29622 raft_consensus.cc:2802] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:36.427996 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.419133 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.404029 29629 raft_consensus.cc:3058] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
W20250905 19:59:36.423309 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 120980us
I20250905 19:59:36.423913 29604 raft_consensus.cc:491] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:36.434960 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:36.437235 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:36.437333 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.442183 28083 raft_consensus.cc:2953] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:36.469240 28050 leader_election.cc:304] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:36.456629 28081 raft_consensus.cc:2466] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:36.426220 29626 raft_consensus.cc:3058] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:36.423444 29685 raft_consensus.cc:3058] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:36.408730 29590 raft_consensus.cc:3058] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:36.429194 29609 raft_consensus.cc:2802] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:36.465469 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.472726 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:36.460877 29714 raft_consensus.cc:3058] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:36.443383 29511 leader_election.cc:290] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:36.454777 29508 raft_consensus.cc:491] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:36.491010 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.491362 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:36.473333 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 164177us
I20250905 19:59:36.463968 29622 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:36.505885 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.474053 29514 leader_election.cc:290] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:36.471911 29604 raft_consensus.cc:3058] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:36.514892 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.471695 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.479321 29627 raft_consensus.cc:2747] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 19:59:36.529069 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.525986 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.530385 28085 raft_consensus.cc:2391] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 2: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:36.504920 29508 raft_consensus.cc:3058] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:36.528688 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:36.512847 29622 raft_consensus.cc:3058] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:36.539844 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:59:36.525977 29559 leader_election.cc:290] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:36.543277 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:59:36.546989 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 409791us
I20250905 19:59:36.585573 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:36.587255 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 74835us
I20250905 19:59:36.591485 28080 raft_consensus.cc:2391] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 3: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:36.590003 27973 leader_election.cc:304] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:36.600618 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:36.605466 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 141842us
I20250905 19:59:36.606792 27975 leader_election.cc:304] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:36.602753 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:36.611032 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 61522.1us
I20250905 19:59:36.611393 29737 raft_consensus.cc:2747] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:36.611586 29735 raft_consensus.cc:2802] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:36.611850 27973 leader_election.cc:304] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:36.612113 29735 raft_consensus.cc:491] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:36.612536 29735 raft_consensus.cc:3058] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:36.612694 29680 raft_consensus.cc:2802] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:36.613063 27973 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:36.613164 29680 raft_consensus.cc:491] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:36.614382 27973 leader_election.cc:304] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:36.543999 29574 raft_consensus.cc:513] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:36.621577 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.623962 27973 leader_election.cc:304] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:36.543386 29713 consensus_queue.cc:1046] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20250905 19:59:36.613809 29680 raft_consensus.cc:3058] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:36.638868 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:36.582257 29732 consensus_queue.cc:1046] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.075s
I20250905 19:59:36.628345 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
W20250905 19:59:36.652195 29717 consensus_meta.cc:287] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.503s user 0.006s sys 0.004s
I20250905 19:59:36.652741 29717 raft_consensus.cc:513] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:36.624969 29618 raft_consensus.cc:2802] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 19:59:36.653728 29618 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:36.654161 29618 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:36.627669 29737 raft_consensus.cc:2747] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 19:59:36.654068 29609 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:36.618968 29742 consensus_queue.cc:1046] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20250905 19:59:36.650215 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:36.655067 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.663492 29609 raft_consensus.cc:3058] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:36.706305 28122 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:36.671236 28748 raft_consensus.cc:1238] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:36.712858 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:36.713691 29710 consensus_meta.cc:287] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.502s user 0.007s sys 0.004s
I20250905 19:59:36.714116 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:36.752148 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 78247.8us
I20250905 19:59:36.683187 29737 raft_consensus.cc:2747] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:36.761978 27975 leader_election.cc:304] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:36.713796 29575 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:36.772377 29575 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:36.715134 29693 raft_consensus.cc:2802] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:36.773854 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.767632 28077 raft_consensus.cc:2391] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 2: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:36.775925 28081 raft_consensus.cc:2466] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 2.
I20250905 19:59:36.766134 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:36.761448 29717 leader_election.cc:290] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:36.794855 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.775406 29693 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:36.714198 29710 raft_consensus.cc:513] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:36.807420 27975 leader_election.cc:304] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a
W20250905 19:59:36.804940 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 95628.6us
I20250905 19:59:36.812887 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:36.816715 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 192831us
I20250905 19:59:36.817600 27973 leader_election.cc:304] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:36.832234 28079 raft_consensus.cc:1238] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:36.815591 29574 leader_election.cc:290] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:36.813495 29693 raft_consensus.cc:3058] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:36.804566 29745 raft_consensus.cc:3053] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:36.856681 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.859035 27973 leader_election.cc:304] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
W20250905 19:59:36.834374 29568 consensus_meta.cc:287] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.507s user 0.010s sys 0.002s
I20250905 19:59:36.826614 29575 raft_consensus.cc:2747] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:36.835278 27975 leader_election.cc:304] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:36.834909 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:36.878212 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 194083us
I20250905 19:59:36.885062 28122 leader_election.cc:304] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:36.892306 27973 leader_election.cc:304] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:36.893251 28715 leader_election.cc:304] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 19:59:36.894831 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 59672.9us
I20250905 19:59:36.893772 27973 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:36.893795 28122 leader_election.cc:304] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:36.865355 29745 raft_consensus.cc:738] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:36.828208 29716 raft_consensus.cc:513] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:36.900453 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 281529us
I20250905 19:59:36.903038 28717 leader_election.cc:304] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:36.908712 27973 leader_election.cc:304] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:36.866885 29751 raft_consensus.cc:2802] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:36.913337 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 19:59:36.901432 29512 consensus_meta.cc:287] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.740s user 0.011s sys 0.001s
I20250905 19:59:36.920315 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.895020 29559 raft_consensus.cc:2747] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:36.898131 29754 raft_consensus.cc:2802] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:36.898218 29574 raft_consensus.cc:2747] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 19:59:36.926868 28751 raft_consensus.cc:1238] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:36.925813 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.879534 29568 raft_consensus.cc:513] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:36.931833 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.901212 29737 raft_consensus.cc:2802] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:36.901026 29713 raft_consensus.cc:2802] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:36.901029 29599 raft_consensus.cc:3058] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:36.901126 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.928563 28084 raft_consensus.cc:2391] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 4: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:36.886503 29748 raft_consensus.cc:2802] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:36.872625 29688 raft_consensus.cc:513] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:36.914279 29751 raft_consensus.cc:491] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:37.014977 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:36.943550 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:37.017370 28156 raft_consensus.cc:1238] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:36.925091 29744 raft_consensus.cc:2802] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:36.929275 29754 raft_consensus.cc:491] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
W20250905 19:59:37.010296 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 114349us
I20250905 19:59:36.907446 29746 consensus_queue.cc:1046] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20250905 19:59:37.021842 27975 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:36.934927 29737 raft_consensus.cc:491] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:36.937072 29713 raft_consensus.cc:491] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:37.042961 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:37.058691 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 150073us
I20250905 19:59:36.941223 29574 raft_consensus.cc:2747] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:36.923508 29590 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:37.052541 29568 leader_election.cc:290] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:37.053224 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:37.014820 29751 raft_consensus.cc:3058] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:37.020592 29745 consensus_queue.cc:260] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:37.020062 29744 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:37.073395 29745 raft_consensus.cc:3058] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
W20250905 19:59:37.054328 29622 consensus_meta.cc:287] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.506s user 0.004s sys 0.004s
I20250905 19:59:36.930205 29512 raft_consensus.cc:513] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:37.068559 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:37.065518 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:37.068900 28715 leader_election.cc:304] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:37.066114 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:37.065023 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:37.066623 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:37.063310 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:37.041666 29754 raft_consensus.cc:3058] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:37.059932 27973 leader_election.cc:304] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:37.043431 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:37.058694 29737 raft_consensus.cc:3058] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
W20250905 19:59:37.070330 29690 consensus_meta.cc:287] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.705s user 0.005s sys 0.010s
W20250905 19:59:37.071668 29644 consensus_meta.cc:287] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.847s user 0.011s sys 0.004s
I20250905 19:59:37.062330 29713 raft_consensus.cc:3058] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:37.011914 29748 raft_consensus.cc:491] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
W20250905 19:59:37.074508 29741 log.cc:927] Time spent T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.305s user 0.002s sys 0.000s
I20250905 19:59:37.020433 29750 raft_consensus.cc:2747] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:37.064231 29710 leader_election.cc:290] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:37.110692 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 207736us
I20250905 19:59:37.127355 28717 leader_election.cc:304] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:37.129200 28717 leader_election.cc:304] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
W20250905 19:59:37.113615 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 53774.7us
I20250905 19:59:37.073421 29744 raft_consensus.cc:3058] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
W20250905 19:59:37.094750 29505 consensus_meta.cc:287] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.940s user 0.011s sys 0.002s
I20250905 19:59:37.087239 29372 raft_consensus.cc:2747] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:37.125383 28159 raft_consensus.cc:1238] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:37.130410 27973 leader_election.cc:304] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:37.086067 29755 raft_consensus.cc:2747] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:37.132787 28717 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:37.117013 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:37.091831 29622 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:37.119546 29748 raft_consensus.cc:3058] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:37.086102 29624 consensus_queue.cc:1046] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20250905 19:59:37.123858 29756 raft_consensus.cc:2802] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:37.145684 27973 leader_election.cc:304] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:37.162593 28122 leader_election.cc:304] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:37.165278 29645 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:37.165964 28122 leader_election.cc:304] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:37.116351 29690 raft_consensus.cc:513] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:37.117599 29644 raft_consensus.cc:513] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:37.135108 29717 raft_consensus.cc:2747] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:37.165422 29754 raft_consensus.cc:513] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:37.170012 29622 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:37.137895 29372 raft_consensus.cc:2747] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:37.170717 28085 raft_consensus.cc:1238] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:37.138198 29505 raft_consensus.cc:513] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:37.173432 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:37.143986 29750 raft_consensus.cc:2747] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:37.177318 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:37.145195 29716 leader_election.cc:290] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:37.172897 29760 raft_consensus.cc:2747] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250905 19:59:37.225410 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 92699.3us
I20250905 19:59:37.168259 29755 raft_consensus.cc:2802] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:37.226516 28749 raft_consensus.cc:1238] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
W20250905 19:59:37.226652 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 52201.4us
I20250905 19:59:37.167151 29756 raft_consensus.cc:491] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:37.231280 28717 leader_election.cc:304] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:37.148389 29710 consensus_queue.cc:1046] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20250905 19:59:37.236063 27973 leader_election.cc:304] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:37.178025 29747 raft_consensus.cc:2747] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:37.240912 28717 leader_election.cc:304] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:37.242214 28717 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:37.245462 27973 leader_election.cc:304] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:37.250396 27973 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:37.251446 28152 raft_consensus.cc:1238] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:37.249436 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 19:59:37.255801 29609 consensus_meta.cc:287] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.543s user 0.005s sys 0.004s
I20250905 19:59:37.256333 29609 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:37.249711 28158 raft_consensus.cc:2391] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 2: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:37.261265 27973 leader_election.cc:304] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:37.259806 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:37.234179 29755 raft_consensus.cc:491] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:37.281070 28718 leader_election.cc:304] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:37.238703 29756 raft_consensus.cc:3058] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:37.248557 29372 raft_consensus.cc:2747] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250905 19:59:37.269748 29680 consensus_meta.cc:287] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.615s user 0.011s sys 0.001s
I20250905 19:59:37.249130 29688 leader_election.cc:290] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:37.250720 29717 raft_consensus.cc:2747] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:37.248730 29750 raft_consensus.cc:2747] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:37.250398 29759 raft_consensus.cc:2747] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:37.268343 29512 leader_election.cc:290] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:37.260218 29590 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:37.277423 29624 raft_consensus.cc:2802] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:37.280913 29450 raft_consensus.cc:2802] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250905 19:59:37.295545 29626 consensus_meta.cc:287] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.791s user 0.006s sys 0.007s
I20250905 19:59:37.285797 29747 raft_consensus.cc:2802] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:37.280072 29755 raft_consensus.cc:3058] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
W20250905 19:59:37.301380 29604 consensus_meta.cc:287] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.766s user 0.013s sys 0.001s
I20250905 19:59:37.324141 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:37.326099 29604 raft_consensus.cc:513] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:37.303085 29609 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:37.332741 29604 leader_election.cc:290] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:37.335444 29693 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:37.337278 29629 consensus_meta.cc:287] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.865s user 0.008s sys 0.002s
I20250905 19:59:37.337720 29629 raft_consensus.cc:513] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:37.335973 28078 raft_consensus.cc:1238] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:37.318341 29450 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:37.337824 29693 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:37.338814 29450 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 19:59:37.339263 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:37.340817 28153 raft_consensus.cc:1238] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:37.300534 29762 raft_consensus.cc:3058] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:37.346628 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 19:59:37.347177 29714 consensus_meta.cc:287] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.833s user 0.013s sys 0.000s
I20250905 19:59:37.301488 29680 raft_consensus.cc:513] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:37.345582 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:37.318715 29624 raft_consensus.cc:491] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:37.321395 29626 raft_consensus.cc:2747] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:37.321287 29747 raft_consensus.cc:491] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:37.363422 29690 leader_election.cc:290] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:37.362461 29644 leader_election.cc:290] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:37.360738 29681 consensus_meta.cc:287] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.988s user 0.005s sys 0.009s
I20250905 19:59:37.358803 29624 raft_consensus.cc:3058] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:37.373275 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:37.373945 29629 leader_election.cc:290] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:37.374986 29754 leader_election.cc:290] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:37.375257 28745 raft_consensus.cc:1238] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:37.371577 29505 leader_election.cc:290] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:37.347581 29714 raft_consensus.cc:513] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:37.361974 29747 raft_consensus.cc:3058] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:37.364410 29688 consensus_queue.cc:1046] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20250905 19:59:37.391883 29740 raft_consensus.cc:3053] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:37.392426 29740 raft_consensus.cc:738] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:37.428550 28079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1246) took 1120 ms. Trace:
I20250905 19:59:37.429795 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:37.431398 28007 raft_consensus.cc:2391] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 2: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:37.431188 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:37.433014 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:37.441864 28752 raft_consensus.cc:1238] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:37.427758 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:37.454624 29738 raft_consensus.cc:3053] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:59:37.455365 29738 raft_consensus.cc:738] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:37.446799 28009 raft_consensus.cc:1238] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:37.452050 28156 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 1810) took 1076 ms. Trace:
I20250905 19:59:37.445151 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:37.447856 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 19:59:37.448459 29508 consensus_meta.cc:287] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.905s user 0.012s sys 0.009s
I20250905 19:59:37.474699 29508 raft_consensus.cc:513] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:37.477916 29680 leader_election.cc:290] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:37.480058 28749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40918 (request call id 383) took 1209 ms. Trace:
I20250905 19:59:37.376689 29681 raft_consensus.cc:513] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:37.480682 28749 rpcz_store.cc:276] 0905 19:59:36.270349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:36.270574 (+ 225us) service_pool.cc:225] Handling call
0905 19:59:36.285983 (+ 15409us) spinlock_profiling.cc:243] Waited 5.3 ms on lock 0x7b5800304598. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b91f60 00007fe4a2b7d1a9 00007fe4a2b7cc52 00007fe49c202b44
0905 19:59:36.286257 (+ 274us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:37.234754 (+948497us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:37.479343 (+244589us) spinlock_profiling.cc:243] Waited 940 ms on lock 0x7b54004b0d54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:37.479761 (+ 418us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2173867520}
I20250905 19:59:37.481467 28080 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868"
dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
mode: GRACEFUL
new_leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:51630
I20250905 19:59:37.482722 29733 raft_consensus.cc:3053] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:37.480126 28077 raft_consensus.cc:2391] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 4: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
W20250905 19:59:37.449682 29599 consensus_meta.cc:287] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.504s user 0.012s sys 0.003s
W20250905 19:59:37.451290 29618 consensus_meta.cc:287] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.796s user 0.011s sys 0.001s
I20250905 19:59:37.442406 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:37.486552 28084 raft_consensus.cc:1238] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:37.437986 28049 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:37.488756 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:37.429546 28079 rpcz_store.cc:276] 0905 19:59:36.307804 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:36.307968 (+ 164us) service_pool.cc:225] Handling call
0905 19:59:36.308265 (+ 297us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:36.832901 (+524636us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:37.427620 (+594719us) spinlock_profiling.cc:243] Waited 524 ms on lock 0x7b54001f09d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:37.428195 (+ 575us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1204418944}
I20250905 19:59:37.495286 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:37.497051 29743 raft_consensus.cc:3053] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:37.497817 29743 raft_consensus.cc:738] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:37.499078 28083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1220) took 1093 ms. Trace:
I20250905 19:59:37.499480 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:37.499711 28748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40918 (request call id 380) took 1291 ms. Trace:
I20250905 19:59:37.500243 28748 rpcz_store.cc:276] 0905 19:59:36.208638 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:36.211883 (+ 3245us) service_pool.cc:225] Handling call
0905 19:59:36.284751 (+ 72868us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:36.712162 (+427411us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:37.499170 (+787008us) spinlock_profiling.cc:243] Waited 371 ms on lock 0x7b54001568d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:37.499535 (+ 365us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":852750592}
I20250905 19:59:37.500926 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:37.501212 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:37.502072 28001 raft_consensus.cc:2391] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 2: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:37.502025 28006 raft_consensus.cc:2466] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 19:59:37.460458 28156 rpcz_store.cc:276] 0905 19:59:36.375906 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:36.376105 (+ 199us) service_pool.cc:225] Handling call
0905 19:59:36.376435 (+ 330us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:37.017998 (+641563us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:37.451314 (+433316us) spinlock_profiling.cc:243] Waited 641 ms on lock 0x7b54000ed3d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:37.451794 (+ 480us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1473369984}
I20250905 19:59:37.502555 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
W20250905 19:59:37.507684 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:37.503134 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:59:37.448573 28158 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39586 (request call id 136) took 1180 ms (client timeout 1507 ms). Trace:
I20250905 19:59:37.504051 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:37.509474 28004 raft_consensus.cc:2466] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 19:59:37.504509 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:37.510874 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:37.511387 28003 raft_consensus.cc:2391] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 2: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:37.511682 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:37.512472 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:37.483577 29681 leader_election.cc:290] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:37.515904 28078 raft_consensus.cc:1238] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:37.513164 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:37.483974 28080 raft_consensus.cc:604] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Received request to transfer leadership to TS cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:37.512524 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:37.523315 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:37.521258 29754 consensus_queue.cc:1046] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20250905 19:59:37.527141 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:37.489053 29733 raft_consensus.cc:738] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:37.536449 28081 raft_consensus.cc:2391] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 3: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:37.457808 29748 raft_consensus.cc:513] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:37.457293 29738 consensus_queue.cc:260] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:37.548354 28122 leader_election.cc:304] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:37.548856 29754 raft_consensus.cc:3053] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:37.566618 29754 raft_consensus.cc:738] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:37.550768 28752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40918 (request call id 384) took 1159 ms. Trace:
W20250905 19:59:37.509192 28158 rpcz_store.cc:269] 0905 19:59:36.267622 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:36.268042 (+ 420us) service_pool.cc:225] Handling call
0905 19:59:37.447937 (+1179895us) spinlock_profiling.cc:243] Waited 904 ms on lock 0x7b540009d3d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:37.448215 (+ 278us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2079155456}
I20250905 19:59:37.555490 29748 leader_election.cc:290] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:37.556061 29599 raft_consensus.cc:2747] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:37.557628 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:37.557976 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:37.559291 28006 raft_consensus.cc:1238] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:37.561268 28003 raft_consensus.cc:2391] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 2: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:37.561190 28009 raft_consensus.cc:1273] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250905 19:59:37.564000 28008 raft_consensus.cc:1238] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:37.572014 28077 raft_consensus.cc:1238] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:37.574651 29622 raft_consensus.cc:3053] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:37.521199 29758 raft_consensus.cc:2953] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:37.499601 28083 rpcz_store.cc:276] 0905 19:59:36.405966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:36.406096 (+ 130us) service_pool.cc:225] Handling call
0905 19:59:36.441696 (+ 35600us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:36.442022 (+ 326us) raft_consensus.cc:1532] Early marking committed up to index 3
0905 19:59:36.507082 (+ 65060us) log.cc:844] Serialized 14 byte log entry
0905 19:59:36.590615 (+ 83533us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:36.590671 (+ 56us) raft_consensus.cc:1643] Marking committed up to 3
0905 19:59:36.590712 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:36.590776 (+ 64us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:36.891097 (+300321us) spinlock_profiling.cc:243] Waited 8.57 ms on lock 0x7b18003c5f80. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49ac723d6 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44
0905 19:59:37.391205 (+500108us) spinlock_profiling.cc:243] Waited 4.09 ms on lock 0x7b18003c5f80. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49aca43e2 00007fe49aca67b1 00007fe49aca6769 00007fe49aca66f9 00007fe49aca66c1 00007fe49aca59bd 0000000000379254 0000000000379099 00007fe49ac8776a
0905 19:59:37.498704 (+107499us) spinlock_profiling.cc:243] Waited 19 ms on lock 0x7b540036db50. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:37.498903 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1240,"raft.queue_time_us":105,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":12143,"spinlock_wait_cycles":72755968,"thread_start_us":70289,"threads_started":1,"wal-append.queue_time_us":112145}
I20250905 19:59:37.565804 28748 raft_consensus.cc:1238] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:37.544963 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:37.583600 28001 raft_consensus.cc:2466] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:37.503110 29743 consensus_queue.cc:260] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:37.585857 29714 leader_election.cc:290] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:37.568933 28752 rpcz_store.cc:276] 0905 19:59:36.391679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:36.391837 (+ 158us) service_pool.cc:225] Handling call
0905 19:59:36.402157 (+ 10320us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:37.442842 (+1040685us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:37.542342 (+ 99500us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b540049d4d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:37.550668 (+ 8326us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2389206272}
I20250905 19:59:37.575207 29622 raft_consensus.cc:738] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:37.578966 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:37.579741 29514 raft_consensus.cc:2747] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 19:59:37.594027 29622 consensus_queue.cc:260] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:37.595697 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:37.596052 29622 raft_consensus.cc:3058] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:37.597922 29624 raft_consensus.cc:513] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:37.537762 29618 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:37.545996 29508 leader_election.cc:290] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:37.589462 28123 leader_election.cc:304] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
W20250905 19:59:37.590121 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 71928.4us
I20250905 19:59:37.590582 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 19:59:37.591624 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 154228us
I20250905 19:59:37.592381 28005 raft_consensus.cc:2391] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 2: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:37.568002 29733 consensus_queue.cc:260] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:37.618278 29733 raft_consensus.cc:3058] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
W20250905 19:59:37.593565 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 45745.7us
I20250905 19:59:37.594369 28749 raft_consensus.cc:3058] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:37.603011 28152 raft_consensus.cc:1238] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:37.606670 29757 raft_consensus.cc:3058] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
W20250905 19:59:37.608392 29735 consensus_meta.cc:287] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.995s user 0.013s sys 0.006s
I20250905 19:59:37.574265 28744 raft_consensus.cc:2953] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:37.568468 29754 consensus_queue.cc:260] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:37.609326 29744 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:37.527848 29740 consensus_queue.cc:260] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:37.611164 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:37.612763 29645 raft_consensus.cc:2747] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:37.613031 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:37.613709 29599 raft_consensus.cc:491] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:37.614643 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:37.611179 29618 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:37.616876 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:37.621479 29753 raft_consensus.cc:2747] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:37.635464 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:59:37.616662 29685 consensus_meta.cc:287] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 1.098s user 0.010s sys 0.009s
I20250905 19:59:37.632126 29507 consensus_queue.cc:1046] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20250905 19:59:37.634702 29619 consensus_queue.cc:1046] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20250905 19:59:37.634474 29599 raft_consensus.cc:513] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:37.620919 29624 leader_election.cc:290] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:37.634511 29749 consensus_queue.cc:1046] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20250905 19:59:37.644547 28082 raft_consensus.cc:2466] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:37.648458 29742 consensus_queue.cc:1046] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250905 19:59:37.649089 29749 raft_consensus.cc:3053] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:37.649623 29749 raft_consensus.cc:738] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:37.650066 29742 raft_consensus.cc:3053] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:59:37.650585 29742 raft_consensus.cc:738] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:37.638402 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:37.634279 28085 raft_consensus.cc:2953] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:37.640115 28010 raft_consensus.cc:2391] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:37.638406 29511 consensus_queue.cc:1046] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20250905 19:59:37.653110 28003 raft_consensus.cc:2391] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:37.644513 29619 raft_consensus.cc:3053] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:37.644927 29711 raft_consensus.cc:3053] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:37.646941 29714 consensus_queue.cc:1046] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250905 19:59:37.647809 29599 leader_election.cc:290] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:37.653344 28749 raft_consensus.cc:2466] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:37.650899 29749 consensus_queue.cc:260] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:37.656159 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:37.663365 29727 consensus_queue.cc:1046] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20250905 19:59:37.665059 29714 raft_consensus.cc:3053] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:37.683707 29714 raft_consensus.cc:738] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:37.684047 29505 consensus_queue.cc:1046] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:37.684580 29727 raft_consensus.cc:3053] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:37.685443 29727 raft_consensus.cc:738] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:37.664063 29711 raft_consensus.cc:738] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:37.661736 29619 raft_consensus.cc:738] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:37.633877 29735 raft_consensus.cc:513] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:37.696193 29619 consensus_queue.cc:260] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:37.697237 29735 leader_election.cc:290] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:37.665825 29644 consensus_queue.cc:1046] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20250905 19:59:37.669426 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:37.689883 29711 consensus_queue.cc:260] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:37.670476 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:37.672786 29508 raft_consensus.cc:3053] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:37.662773 29685 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:37.667872 29742 consensus_queue.cc:260] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:37.686839 29693 raft_consensus.cc:3053] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:37.687667 28123 leader_election.cc:304] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 19:59:37.665962 29627 consensus_queue.cc:1046] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20250905 19:59:37.703676 29508 raft_consensus.cc:738] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:37.705256 29693 raft_consensus.cc:738] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:37.703810 28003 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:59:37.706132 29508 consensus_queue.cc:260] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:37.706485 29693 consensus_queue.cc:260] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:37.707244 28752 raft_consensus.cc:1238] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:37.707648 29508 raft_consensus.cc:3058] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:37.707919 29693 raft_consensus.cc:3058] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:37.685328 29714 consensus_queue.cc:260] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:37.709677 29714 raft_consensus.cc:3058] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:37.710762 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:37.692835 29743 raft_consensus.cc:3058] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:37.712983 29685 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:37.701721 29754 raft_consensus.cc:3058] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:37.702571 29711 raft_consensus.cc:3058] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:37.703032 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:37.703291 28006 raft_consensus.cc:1238] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
W20250905 19:59:37.716581 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:37.698029 29619 raft_consensus.cc:3058] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:37.716775 29627 raft_consensus.cc:3053] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:37.717568 29742 raft_consensus.cc:3058] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Advancing to term 4
W20250905 19:59:37.720876 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:37.715082 29768 raft_consensus.cc:3053] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Stepping down as leader of term 1
W20250905 19:59:37.727178 29668 consensus_meta.cc:287] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 1.167s user 0.007s sys 0.012s
I20250905 19:59:37.727429 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:37.738677 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:37.739012 29756 raft_consensus.cc:513] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:37.690455 29727 consensus_queue.cc:260] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:37.740128 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:37.740469 27905 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34440 (request call id 250) took 1560 ms. Trace:
I20250905 19:59:37.722934 29627 raft_consensus.cc:738] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:37.741874 29727 raft_consensus.cc:3058] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:37.729954 29768 raft_consensus.cc:738] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:37.742030 29627 consensus_queue.cc:260] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:37.745564 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:37.748193 29740 raft_consensus.cc:3058] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:37.744614 29768 consensus_queue.cc:260] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:37.753638 28004 raft_consensus.cc:2466] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
I20250905 19:59:37.753178 29768 raft_consensus.cc:3058] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:37.744616 29627 raft_consensus.cc:3058] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:37.760857 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:37.752578 29645 raft_consensus.cc:2802] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:37.740933 27905 rpcz_store.cc:276] 0905 19:59:36.180206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:36.182554 (+ 2348us) service_pool.cc:225] Handling call
0905 19:59:37.740199 (+1557645us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:59:36.534692 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:59:36.668138 (+133446us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:59:36.672172 (+ 4034us) write_op.cc:435] Acquired schema lock
0905 19:59:36.673560 (+ 1388us) tablet.cc:660] Decoding operations
0905 19:59:36.804419 (+130859us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:59:36.810330 (+ 5911us) write_op.cc:641] Partition lock acquired for write op
0905 19:59:36.813678 (+ 3348us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:59:36.864494 (+ 50816us) tablet.cc:699] Row locks acquired
0905 19:59:36.866213 (+ 1719us) write_op.cc:260] PREPARE: finished
0905 19:59:36.889095 (+ 22882us) write_op.cc:270] Start()
0905 19:59:36.899766 (+ 10671us) write_op.cc:276] Timestamp: P: 1757102376883458 usec, L: 0
0905 19:59:36.901235 (+ 1469us) op_driver.cc:348] REPLICATION: starting
0905 19:59:37.048902 (+147667us) log.cc:844] Serialized 1045 byte log entry
0905 19:59:37.396321 (+347419us) op_driver.cc:464] REPLICATION: finished
0905 19:59:37.463649 (+ 67328us) write_op.cc:301] APPLY: starting
0905 19:59:37.463868 (+ 219us) tablet.cc:1365] starting BulkCheckPresence
0905 19:59:37.464143 (+ 275us) tablet.cc:1368] finished BulkCheckPresence
0905 19:59:37.506727 (+ 42584us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:59:37.589431 (+ 82704us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:59:37.589910 (+ 479us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0905 19:59:37.589985 (+ 75us) write_op.cc:312] APPLY: finished
0905 19:59:37.590394 (+ 409us) log.cc:844] Serialized 29 byte log entry
0905 19:59:37.590725 (+ 331us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:59:37.590918 (+ 193us) write_op.cc:454] Released schema lock
0905 19:59:37.590974 (+ 56us) write_op.cc:341] FINISH: Updating metrics
0905 19:59:37.591660 (+ 686us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:59:37.591689 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":367,"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":66666,"apply.run_cpu_time_us":4970,"apply.run_wall_time_us":128141,"num_ops":2,"prepare.queue_time_us":256254,"prepare.run_cpu_time_us":22812,"prepare.run_wall_time_us":781154,"raft.queue_time_us":25405,"raft.run_cpu_time_us":1834,"raft.run_wall_time_us":174521,"replication_time_us":494917,"thread_start_us":454116,"threads_started":4,"wal-append.queue_time_us":282504}]]}
I20250905 19:59:37.764146 28003 raft_consensus.cc:2466] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 19:59:37.764459 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:37.766554 28003 raft_consensus.cc:1238] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20250905 19:59:37.762975 28122 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:37.752424 29668 raft_consensus.cc:513] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:37.774610 29738 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:37.775089 29762 raft_consensus.cc:2747] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:37.775575 29505 raft_consensus.cc:2747] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 19:59:37.770627 29749 raft_consensus.cc:3058] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:37.772552 29645 raft_consensus.cc:695] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:37.774005 29744 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:37.790742 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:37.790872 28002 raft_consensus.cc:1238] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
W20250905 19:59:37.793699 29751 consensus_meta.cc:287] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.700s user 0.003s sys 0.012s
I20250905 19:59:37.796111 28748 raft_consensus.cc:1238] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:37.798750 29755 raft_consensus.cc:513] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:37.798750 29756 leader_election.cc:290] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:37.803059 29737 consensus_meta.cc:287] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.684s user 0.016s sys 0.000s
W20250905 19:59:37.804387 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:37.794342 29751 raft_consensus.cc:513] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:37.819769 28082 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1247) took 1175 ms (client timeout 1519 ms). Trace:
I20250905 19:59:37.819908 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 19:59:37.820533 28082 rpcz_store.cc:269] 0905 19:59:36.644819 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:36.649768 (+ 4949us) service_pool.cc:225] Handling call
0905 19:59:37.818144 (+1168376us) spinlock_profiling.cc:243] Waited 782 ms on lock 0x7b54003bf6d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:37.819685 (+ 1541us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1798043520}
I20250905 19:59:37.803536 29737 raft_consensus.cc:513] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:37.823915 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:37.825484 28003 raft_consensus.cc:2391] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 2: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:37.830798 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:37.835008 28007 raft_consensus.cc:2391] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 2: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:37.836542 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 19:59:37.829344 29713 consensus_meta.cc:287] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.650s user 0.010s sys 0.005s
I20250905 19:59:37.838343 28002 raft_consensus.cc:1238] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 1. Current term is 2. Ops: []
W20250905 19:59:37.840955 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 49092.3us
W20250905 19:59:37.841693 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:37.844244 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:37.846582 28003 raft_consensus.cc:2391] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 2: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:37.847581 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:37.850070 28084 raft_consensus.cc:1238] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:37.853298 29747 raft_consensus.cc:513] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:37.854143 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:37.853729 28076 raft_consensus.cc:1238] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:37.856307 28005 raft_consensus.cc:1238] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:37.856703 28010 raft_consensus.cc:1238] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:37.862905 28004 raft_consensus.cc:1238] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:37.865487 28007 raft_consensus.cc:1238] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:37.859134 29668 leader_election.cc:290] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:37.846284 29713 raft_consensus.cc:513] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:37.872982 28002 raft_consensus.cc:1238] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:37.875385 28007 raft_consensus.cc:1238] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:37.869575 29626 consensus_queue.cc:1046] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20250905 19:59:37.863819 29755 leader_election.cc:290] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:37.878046 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:37.886307 28007 raft_consensus.cc:1238] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:37.886731 28010 raft_consensus.cc:1238] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:37.891628 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:37.882985 28748 raft_consensus.cc:1238] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 1. Current term is 2. Ops: []
W20250905 19:59:37.896979 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:37.862949 29645 consensus_queue.cc:237] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:37.888283 29751 leader_election.cc:290] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:37.898902 28002 raft_consensus.cc:1238] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: []
I20250905 19:59:37.901996 28151 raft_consensus.cc:1238] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:37.902921 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:37.889938 29737 leader_election.cc:290] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:37.905408 28005 raft_consensus.cc:1238] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:37.910646 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:37.913331 29644 raft_consensus.cc:3053] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:37.913728 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:37.913848 29644 raft_consensus.cc:738] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:37.916018 29644 consensus_queue.cc:260] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:37.919191 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:37.920579 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:37.921018 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:37.922415 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:37.925171 28158 raft_consensus.cc:2466] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 3.
I20250905 19:59:37.925586 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:37.926504 28051 leader_election.cc:304] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:37.927958 28151 raft_consensus.cc:2466] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:37.928221 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:37.935858 29757 raft_consensus.cc:2747] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:37.939559 28010 raft_consensus.cc:1238] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:37.971758 29727 raft_consensus.cc:3058] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:37.972229 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:37.973373 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:37.940099 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:37.974452 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:37.974803 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:37.975016 28010 raft_consensus.cc:3058] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:37.975857 28005 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0"
dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
mode: GRACEFUL
new_leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
from {username='slave'} at 127.0.0.1:47564
I20250905 19:59:37.976444 28005 raft_consensus.cc:604] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Received request to transfer leadership to TS e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:37.977875 28005 raft_consensus.cc:1238] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:37.941048 28007 raft_consensus.cc:1238] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 2. Current term is 3. Ops: []
W20250905 19:59:37.978928 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 52683.6us
I20250905 19:59:37.979230 27975 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:37.980989 29626 consensus_queue.cc:1046] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:37.982264 29511 raft_consensus.cc:3053] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:37.982735 29511 raft_consensus.cc:738] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:37.937839 29713 leader_election.cc:290] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:37.942689 29768 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:37.945777 29644 raft_consensus.cc:3058] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:37.946084 28752 raft_consensus.cc:1238] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:37.937839 28085 raft_consensus.cc:2466] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 3.
I20250905 19:59:37.947063 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:37.947322 28002 raft_consensus.cc:1238] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:37.948335 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:37.948843 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 19:59:37.949786 29450 consensus_meta.cc:287] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.610s user 0.013s sys 0.005s
I20250905 19:59:37.914968 29747 leader_election.cc:290] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:37.951609 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:37.951630 28008 raft_consensus.cc:2466] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:37.957273 28006 raft_consensus.cc:2466] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:37.958710 28004 raft_consensus.cc:1238] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 4. Current term is 5. Ops: []
I20250905 19:59:37.969493 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:38.007176 28005 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614"
dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
mode: GRACEFUL
new_leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
from {username='slave'} at 127.0.0.1:47564
I20250905 19:59:38.007875 28005 raft_consensus.cc:604] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Received request to transfer leadership to TS 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:38.008244 28005 raft_consensus.cc:612] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250905 19:59:37.992259 29737 raft_consensus.cc:2747] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250905 19:59:38.012835 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:38.013108 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:38.011432 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:38.012207 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:38.014928 28084 raft_consensus.cc:1238] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:38.013464 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:38.016353 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:38.018743 28157 raft_consensus.cc:1238] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:38.019501 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:38.019832 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250905 19:59:38.020759 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:38.001137 29733 consensus_queue.cc:1046] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20250905 19:59:38.007365 29768 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:38.025651 29768 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:38.026888 28125 leader_election.cc:304] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.004537 29450 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.029207 28051 leader_election.cc:304] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:38.030589 29450 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:38.031033 28156 raft_consensus.cc:1238] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:38.031311 28001 raft_consensus.cc:1238] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 5. Current term is 6. Ops: []
I20250905 19:59:38.030927 29511 consensus_queue.cc:260] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:37.993165 29668 consensus_queue.cc:1046] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20250905 19:59:38.032045 29511 raft_consensus.cc:3058] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:38.033079 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:38.034420 28077 raft_consensus.cc:1238] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:38.034797 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:38.035583 28155 raft_consensus.cc:2391] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 2: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:38.037153 28745 raft_consensus.cc:2466] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
I20250905 19:59:38.038064 29626 consensus_queue.cc:1046] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:38.039317 29626 raft_consensus.cc:3058] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:38.041761 29711 raft_consensus.cc:3053] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:38.042191 29711 raft_consensus.cc:738] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:38.043602 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:59:38.046900 28049 leader_election.cc:304] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:38.045186 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:59:38.052229 28010 raft_consensus.cc:2466] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 3.
I20250905 19:59:38.053138 29619 raft_consensus.cc:3058] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:38.053211 29768 raft_consensus.cc:2802] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:38.055369 29768 raft_consensus.cc:695] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:38.058416 29768 consensus_queue.cc:237] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:38.065104 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 72980.7us
W20250905 19:59:38.067009 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:38.067426 29745 raft_consensus.cc:3053] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:38.067849 29745 raft_consensus.cc:738] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:38.068152 29727 raft_consensus.cc:2747] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:38.068827 29745 consensus_queue.cc:260] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:38.069998 29763 raft_consensus.cc:3053] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:59:38.073550 28125 leader_election.cc:304] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.076526 28051 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:38.075256 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:38.070329 29745 raft_consensus.cc:3058] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:38.084128 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:38.090296 28717 leader_election.cc:304] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.073663 29763 raft_consensus.cc:738] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:38.048390 29711 consensus_queue.cc:260] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.093647 27975 leader_election.cc:304] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:38.084234 29758 raft_consensus.cc:2747] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250905 19:59:38.096100 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 111649us
I20250905 19:59:38.098943 29742 raft_consensus.cc:2802] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:38.099697 29742 raft_consensus.cc:695] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 3 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
W20250905 19:59:38.100087 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 43498.6us
I20250905 19:59:38.100616 29742 consensus_queue.cc:237] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:38.101352 28051 leader_election.cc:304] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:38.088865 29727 raft_consensus.cc:2747] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 19:59:38.107816 29508 raft_consensus.cc:3058] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:38.108021 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:38.110030 28125 leader_election.cc:304] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.114055 28125 leader_election.cc:304] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.115599 28125 leader_election.cc:304] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:38.120564 28009 raft_consensus.cc:2466] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
I20250905 19:59:38.106815 29619 raft_consensus.cc:2747] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:38.116755 29738 raft_consensus.cc:2802] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:38.133973 29738 raft_consensus.cc:695] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:38.134073 29763 consensus_queue.cc:260] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.135077 29738 consensus_queue.cc:237] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.136345 29763 raft_consensus.cc:3058] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:38.121085 29757 raft_consensus.cc:3058] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:38.122352 29693 raft_consensus.cc:2747] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:38.131217 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:38.143389 29758 raft_consensus.cc:2747] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:38.144739 28125 leader_election.cc:304] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.146688 28125 leader_election.cc:304] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.146816 29619 raft_consensus.cc:2802] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:38.147928 29693 raft_consensus.cc:2802] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:38.148600 29693 raft_consensus.cc:491] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:38.149009 29693 raft_consensus.cc:3058] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:38.149406 28125 leader_election.cc:304] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.152210 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:38.154634 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:38.156750 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:38.157291 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:38.159731 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:38.160588 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:38.163741 29711 raft_consensus.cc:3058] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:38.162250 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:59:38.159323 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 47718.4us
W20250905 19:59:38.167121 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:38.159567 29619 raft_consensus.cc:491] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:38.162932 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:38.172595 29645 raft_consensus.cc:3058] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
W20250905 19:59:38.180106 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 78809.6us
I20250905 19:59:38.169749 27905 catalog_manager.cc:5582] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 1 to 2, leader changed from cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 2 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:38.193425 27975 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:38.185881 27973 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.198539 28051 leader_election.cc:304] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:38.203444 28078 raft_consensus.cc:2466] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:38.204169 27975 leader_election.cc:304] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:38.170826 29619 raft_consensus.cc:3058] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
W20250905 19:59:38.208916 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 59699.4us
I20250905 19:59:38.229061 28050 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:38.207994 29450 raft_consensus.cc:2747] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 19:59:38.241852 29737 raft_consensus.cc:2747] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250905 19:59:38.242229 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 143791us
I20250905 19:59:38.230123 29733 raft_consensus.cc:2747] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 19:59:38.209506 29768 raft_consensus.cc:2747] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:38.246981 29757 raft_consensus.cc:2747] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:38.248137 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:38.249581 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:38.249795 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:38.250833 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:38.252043 27973 leader_election.cc:304] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.252854 29626 raft_consensus.cc:2747] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250905 19:59:38.253809 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:38.258375 28083 raft_consensus.cc:2466] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:38.260536 29644 raft_consensus.cc:2747] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:38.261009 29745 consensus_queue.cc:1046] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20250905 19:59:38.266185 29745 raft_consensus.cc:3053] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:38.207049 29693 raft_consensus.cc:513] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.266587 29745 raft_consensus.cc:738] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:38.267412 29745 consensus_queue.cc:260] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.268252 29693 leader_election.cc:290] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:38.268932 29745 raft_consensus.cc:3058] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:38.269007 28125 leader_election.cc:304] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.270486 28125 leader_election.cc:304] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.260167 27973 leader_election.cc:304] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.271739 28125 leader_election.cc:304] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.272627 29713 raft_consensus.cc:2747] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:38.272948 28125 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:38.273177 29758 raft_consensus.cc:2747] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:38.273825 28051 leader_election.cc:304] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:38.273895 29713 consensus_queue.cc:1046] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20250905 19:59:38.274768 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:38.275341 28051 leader_election.cc:304] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:38.275553 29619 raft_consensus.cc:513] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 19:59:38.276180 29511 raft_consensus.cc:2747] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:38.276633 29747 raft_consensus.cc:991] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0: : Instructing follower e4413ae5e0b64b52a6f360ac580239b9 to start an election
I20250905 19:59:38.276690 28051 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:38.276913 29511 raft_consensus.cc:2747] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:38.277202 29747 raft_consensus.cc:1079] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Signalling peer e4413ae5e0b64b52a6f360ac580239b9 to start an election
I20250905 19:59:38.277585 29768 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:38.278071 28051 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:38.272620 29599 raft_consensus.cc:2802] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250905 19:59:38.279299 29777 log.cc:927] Time spent T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.138s user 0.000s sys 0.001s
I20250905 19:59:38.276332 29668 raft_consensus.cc:3053] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:38.283365 29733 raft_consensus.cc:2747] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 19:59:38.283093 29737 consensus_queue.cc:1046] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:38.285113 27973 leader_election.cc:304] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.289848 29727 raft_consensus.cc:3058] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:38.300387 28156 raft_consensus.cc:1238] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:38.303865 29645 raft_consensus.cc:2747] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:38.286213 29505 raft_consensus.cc:2747] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250905 19:59:38.303916 29751 raft_consensus.cc:2747] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 19:59:38.306047 29508 raft_consensus.cc:2747] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:38.287850 29668 raft_consensus.cc:738] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:38.286494 29599 raft_consensus.cc:491] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:38.288611 29693 raft_consensus.cc:2802] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 19:59:38.315620 28051 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:38.315511 27973 leader_election.cc:304] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.321451 27973 leader_election.cc:304] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.312707 29599 raft_consensus.cc:3058] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:38.312819 29693 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:38.328933 29737 raft_consensus.cc:2747] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 19:59:38.319031 29505 consensus_queue.cc:1046] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20250905 19:59:38.318847 29755 raft_consensus.cc:3053] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:38.311110 29622 consensus_queue.cc:1046] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20250905 19:59:38.336428 27973 leader_election.cc:304] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.332391 29711 raft_consensus.cc:2802] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:38.317526 29619 leader_election.cc:290] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:38.333082 29713 raft_consensus.cc:2747] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250905 19:59:38.355279 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 46971.3us
I20250905 19:59:38.333375 29693 raft_consensus.cc:3058] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:38.337206 29755 raft_consensus.cc:738] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:38.360610 27973 leader_election.cc:304] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.356518 28125 leader_election.cc:304] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.349000 29711 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:38.367053 29713 raft_consensus.cc:2747] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 19:59:38.370815 28051 leader_election.cc:304] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:38.359163 29737 raft_consensus.cc:2747] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 19:59:38.369570 27973 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.337796 29668 consensus_queue.cc:260] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.366601 29711 raft_consensus.cc:3058] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:38.383147 29627 raft_consensus.cc:2747] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250905 19:59:38.386045 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:38.373580 29645 raft_consensus.cc:3053] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:38.389392 27973 leader_election.cc:304] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.379313 29755 consensus_queue.cc:260] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:38.398993 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 51650us
I20250905 19:59:38.399886 29738 raft_consensus.cc:2747] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 19:59:38.400681 29668 raft_consensus.cc:3058] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:38.390807 29756 raft_consensus.cc:2747] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250905 19:59:38.385497 29619 raft_consensus.cc:2747] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 19:59:38.403015 27973 leader_election.cc:304] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.370632 29508 consensus_queue.cc:1046] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20250905 19:59:38.391242 29626 raft_consensus.cc:3053] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:38.408008 29768 raft_consensus.cc:2747] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 19:59:38.414173 29626 raft_consensus.cc:738] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:38.418241 29644 raft_consensus.cc:3053] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:38.395066 29755 raft_consensus.cc:3058] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:38.418864 29644 raft_consensus.cc:738] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:38.419888 29644 consensus_queue.cc:260] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.421118 29644 raft_consensus.cc:3058] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:38.430898 28051 leader_election.cc:304] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:38.432093 29622 raft_consensus.cc:2802] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:38.432797 29622 raft_consensus.cc:695] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:38.432956 28051 leader_election.cc:304] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:38.433897 29768 raft_consensus.cc:2747] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 19:59:38.433912 29622 consensus_queue.cc:237] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.390168 29645 raft_consensus.cc:738] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:38.442031 29619 consensus_queue.cc:1046] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:38.431073 29626 consensus_queue.cc:260] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.443450 29619 raft_consensus.cc:3053] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:38.449640 29619 raft_consensus.cc:738] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:38.449720 29727 raft_consensus.cc:2747] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:38.440526 28125 leader_election.cc:304] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.450703 29619 consensus_queue.cc:260] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.452330 29619 raft_consensus.cc:3058] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:38.447170 28152 raft_consensus.cc:1238] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:38.447381 29599 raft_consensus.cc:513] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:38.448992 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 46056.3us
I20250905 19:59:38.442121 29645 consensus_queue.cc:260] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:38.441046 28051 leader_election.cc:304] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:38.452351 29757 raft_consensus.cc:2747] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 19:59:38.455739 29749 consensus_queue.cc:1046] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250905 19:59:38.457022 27973 leader_election.cc:304] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.457134 29757 consensus_queue.cc:1046] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:38.457894 29763 raft_consensus.cc:2747] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 19:59:38.460052 29505 raft_consensus.cc:3053] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:38.460500 29505 raft_consensus.cc:738] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:38.461359 29505 consensus_queue.cc:260] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.465051 29599 leader_election.cc:290] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 19:59:38.469743 28718 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.800s (SENT) [suppressed 49 similar messages]
W20250905 19:59:38.489565 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 55247.9us
I20250905 19:59:38.491055 27973 leader_election.cc:304] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.492636 28051 leader_election.cc:304] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:38.474651 29745 raft_consensus.cc:2747] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 19:59:38.503527 28744 raft_consensus.cc:1273] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250905 19:59:38.504995 29508 raft_consensus.cc:2747] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 19:59:38.508075 27973 leader_election.cc:304] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.509853 27973 leader_election.cc:304] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.510951 29763 raft_consensus.cc:2747] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 19:59:38.511857 29763 raft_consensus.cc:2747] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 19:59:38.513332 28051 leader_election.cc:304] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:38.513569 27973 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.514731 29622 raft_consensus.cc:3058] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:38.522394 29749 consensus_queue.cc:1035] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250905 19:59:38.522866 29758 raft_consensus.cc:3053] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:38.523260 29755 raft_consensus.cc:2747] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 19:59:38.537876 29763 raft_consensus.cc:2747] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 19:59:38.500375 29645 raft_consensus.cc:3058] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:38.566535 29758 raft_consensus.cc:738] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:38.498327 29626 raft_consensus.cc:3058] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:38.533437 29505 raft_consensus.cc:3058] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:38.568202 29787 raft_consensus.cc:491] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:38.493860 29727 raft_consensus.cc:2747] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 19:59:38.543326 27973 leader_election.cc:304] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:38.568835 29787 raft_consensus.cc:513] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.539635 28745 raft_consensus.cc:1273] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250905 19:59:38.572309 29727 consensus_queue.cc:1035] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20250905 19:59:38.523828 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 71426.8us
I20250905 19:59:38.573765 29787 leader_election.cc:290] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:38.573935 29788 consensus_queue.cc:1046] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:38.574635 29763 raft_consensus.cc:2802] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:38.575618 29787 raft_consensus.cc:3053] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:38.576109 29787 raft_consensus.cc:738] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:38.524329 28051 leader_election.cc:304] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:38.577080 29763 raft_consensus.cc:695] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:38.582159 29763 consensus_queue.cc:237] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.582935 29693 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:38.585338 29693 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:38.586141 27973 leader_election.cc:304] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:38.587098 29755 raft_consensus.cc:2802] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:38.586870 29757 consensus_queue.cc:1046] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:38.587625 29755 raft_consensus.cc:695] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:38.588158 29749 raft_consensus.cc:3053] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:38.588573 29749 raft_consensus.cc:738] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:38.589409 29749 consensus_queue.cc:260] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.589933 29742 consensus_queue.cc:1046] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250905 19:59:38.591068 28718 leader_election.cc:304] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:38.592585 29714 raft_consensus.cc:2747] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 19:59:38.591188 29742 raft_consensus.cc:3053] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:38.593834 29742 raft_consensus.cc:738] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:38.593142 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:38.595249 28718 leader_election.cc:304] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.596308 29742 consensus_queue.cc:260] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:38.599757 29742 raft_consensus.cc:3058] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:38.601095 29714 raft_consensus.cc:2747] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 19:59:38.602329 29626 raft_consensus.cc:2747] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 19:59:38.588585 29755 consensus_queue.cc:237] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.604137 29749 raft_consensus.cc:3058] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
W20250905 19:59:38.604995 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 80757.6us
I20250905 19:59:38.542987 29711 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:38.606606 29727 consensus_queue.cc:1046] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:38.607857 29644 raft_consensus.cc:3053] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:38.608325 29644 raft_consensus.cc:738] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:38.609447 29644 consensus_queue.cc:260] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.611188 29644 raft_consensus.cc:3058] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:38.568233 29758 consensus_queue.cc:260] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:38.614821 27908 catalog_manager.cc:5582] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 1 to 2, leader changed from e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 2 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:38.578730 29787 consensus_queue.cc:260] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.618165 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:38.619932 27908 catalog_manager.cc:5582] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: config changed from index 3 to 4, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:38.636845 28156 raft_consensus.cc:1273] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250905 19:59:38.619477 29711 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:38.623724 29787 raft_consensus.cc:3058] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:38.638808 29508 consensus_queue.cc:1035] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250905 19:59:38.615787 29758 raft_consensus.cc:3058] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:38.644513 28078 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0"
dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
from {username='slave'} at 127.0.0.1:59550
I20250905 19:59:38.645184 28078 raft_consensus.cc:491] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:38.645561 28078 raft_consensus.cc:3058] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:38.653285 28078 raft_consensus.cc:513] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.657765 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:38.658645 28158 raft_consensus.cc:3058] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:38.627174 29619 consensus_queue.cc:1046] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20250905 19:59:38.627444 29757 consensus_queue.cc:1046] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20250905 19:59:38.628399 28718 leader_election.cc:304] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.666919 29757 raft_consensus.cc:3053] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Stepping down as leader of term 4
I20250905 19:59:38.664606 29693 raft_consensus.cc:3053] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:38.631108 28051 leader_election.cc:304] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:38.667936 29693 raft_consensus.cc:738] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:38.668591 28158 raft_consensus.cc:2466] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:38.630486 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:38.668970 29693 consensus_queue.cc:260] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.670377 28050 leader_election.cc:304] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:38.667438 29757 raft_consensus.cc:738] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
W20250905 19:59:38.623153 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:38.627326 29505 consensus_queue.cc:1046] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20250905 19:59:38.671886 29505 raft_consensus.cc:3053] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:38.671743 29757 consensus_queue.cc:260] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.672174 28715 leader_election.cc:304] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:38.672492 29768 raft_consensus.cc:2802] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:38.627740 28125 leader_election.cc:304] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:38.672974 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:38.673071 29768 raft_consensus.cc:695] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
W20250905 19:59:38.674208 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 46566.9us
I20250905 19:59:38.674513 29619 raft_consensus.cc:2747] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:38.674391 29768 consensus_queue.cc:237] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.676192 29508 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:38.676299 29645 consensus_queue.cc:1046] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:38.676891 29619 consensus_queue.cc:1046] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:38.676821 29508 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:38.677392 29645 consensus_queue.cc:1046] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250905 19:59:38.678036 29619 raft_consensus.cc:3053] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:38.670477 29714 raft_consensus.cc:2747] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 19:59:38.678599 29788 raft_consensus.cc:3053] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:38.678996 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:38.679077 29788 raft_consensus.cc:738] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:38.679203 29714 consensus_queue.cc:1046] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250905 19:59:38.680287 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:38.680949 28156 raft_consensus.cc:1238] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:38.680810 29619 raft_consensus.cc:738] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:38.680862 28010 raft_consensus.cc:3053] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:38.678612 29644 raft_consensus.cc:2802] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:38.681964 28010 raft_consensus.cc:738] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:38.682548 29644 raft_consensus.cc:695] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:38.682301 29619 consensus_queue.cc:260] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.682896 28010 consensus_queue.cc:260] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.682907 29711 consensus_queue.cc:1046] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:38.683733 29619 raft_consensus.cc:3058] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:38.683475 29644 consensus_queue.cc:237] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.684232 28010 raft_consensus.cc:3058] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:38.684458 29511 raft_consensus.cc:3053] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:38.684863 29511 raft_consensus.cc:738] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:38.685710 29511 consensus_queue.cc:260] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 19:59:38.686800 29511 raft_consensus.cc:3058] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:38.618073 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:38.687592 29788 consensus_queue.cc:260] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.672283 29505 raft_consensus.cc:738] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:38.689074 29788 raft_consensus.cc:3058] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:38.689440 29505 consensus_queue.cc:260] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.689092 27904 catalog_manager.cc:5582] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 1 to 2, leader changed from cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 2 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:38.690702 29505 raft_consensus.cc:3058] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:38.691186 29749 raft_consensus.cc:3053] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 LEADER]: Stepping down as leader of term 5
I20250905 19:59:38.691592 29749 raft_consensus.cc:738] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:38.692523 28050 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:38.692430 29749 consensus_queue.cc:260] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.693368 29711 raft_consensus.cc:2747] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 19:59:38.693920 28125 leader_election.cc:304] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.672995 29757 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:38.695008 28085 raft_consensus.cc:1238] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: []
I20250905 19:59:38.695214 29645 raft_consensus.cc:2802] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:38.695819 29645 raft_consensus.cc:695] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:38.697978 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:38.698137 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:38.698628 28083 raft_consensus.cc:2374] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3. Re-sending same reply.
I20250905 19:59:38.700212 28085 raft_consensus.cc:1238] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:38.696885 29645 consensus_queue.cc:237] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.704488 29508 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:38.705282 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:38.707726 28152 raft_consensus.cc:2466] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:38.709496 28050 leader_election.cc:304] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:38.713086 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:38.714709 28083 raft_consensus.cc:1238] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:38.715770 28125 leader_election.cc:304] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.719974 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:38.720759 28081 raft_consensus.cc:2391] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 3: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:38.716993 29755 raft_consensus.cc:491] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:38.721622 29599 raft_consensus.cc:2802] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:38.721311 29755 raft_consensus.cc:513] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:38.722056 29599 raft_consensus.cc:491] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:38.722565 29599 raft_consensus.cc:3058] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:38.724018 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:38.724687 29749 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
I20250905 19:59:38.728428 28010 raft_consensus.cc:2466] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:38.728760 28078 leader_election.cc:290] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:38.731595 29755 leader_election.cc:290] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:38.733563 28082 consensus_queue.cc:237] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:38.756040 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:38.625516 29622 raft_consensus.cc:2747] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:38.728562 29626 raft_consensus.cc:2802] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250905 19:59:38.827051 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 154961us
I20250905 19:59:38.775483 29742 raft_consensus.cc:2802] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:38.837610 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:38.841442 28006 raft_consensus.cc:2391] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 2: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
W20250905 19:59:38.838862 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:38.839576 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:38.840180 28079 raft_consensus.cc:1238] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:38.843688 29800 raft_consensus.cc:3053] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:38.844121 29800 raft_consensus.cc:738] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:38.845227 29800 consensus_queue.cc:260] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:38.846372 29800 raft_consensus.cc:3058] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:38.849604 29599 raft_consensus.cc:513] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:38.852452 29599 leader_election.cc:290] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 19:59:38.854024 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:38.855661 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 187129us
I20250905 19:59:38.852896 28085 raft_consensus.cc:2466] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
W20250905 19:59:38.856885 27975 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:38.858832 27973 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.860317 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:38.860814 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:38.861160 28003 raft_consensus.cc:1238] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20250905 19:59:38.833526 29626 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:38.865860 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:38.868100 29626 raft_consensus.cc:3058] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:38.856369 29693 raft_consensus.cc:3058] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:38.882936 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:38.883970 28752 raft_consensus.cc:2466] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:38.866644 29742 raft_consensus.cc:695] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:38.885156 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:38.885346 29742 consensus_queue.cc:237] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:38.886135 28081 raft_consensus.cc:2466] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:38.863423 29738 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:38.891604 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:38.855281 28008 raft_consensus.cc:1273] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250905 19:59:38.895756 28001 raft_consensus.cc:2466] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:38.840065 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 19:59:38.895229 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 167096us
I20250905 19:59:38.908217 27973 leader_election.cc:304] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:38.913795 28152 raft_consensus.cc:1238] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 4. Current term is 5. Ops: []
I20250905 19:59:38.915795 28125 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:38.895131 29713 raft_consensus.cc:2747] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250905 19:59:38.922295 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:38.925109 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:38.930018 29805 raft_consensus.cc:2747] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250905 19:59:38.932294 28125 leader_election.cc:304] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:38.934453 28157 raft_consensus.cc:1273] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250905 19:59:38.913200 29800 consensus_queue.cc:1046] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20250905 19:59:38.939074 28717 leader_election.cc:304] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:38.943544 28749 raft_consensus.cc:1273] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:38.917068 29763 raft_consensus.cc:2802] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:38.950973 28718 leader_election.cc:304] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:38.905731 28745 raft_consensus.cc:2466] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:38.951936 29763 raft_consensus.cc:491] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:38.956609 29787 raft_consensus.cc:2802] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:38.896116 29801 consensus_queue.cc:1046] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.059s
I20250905 19:59:38.897992 29738 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:38.956816 29805 consensus_queue.cc:1035] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20250905 19:59:38.961861 29787 raft_consensus.cc:491] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:38.972427 29787 raft_consensus.cc:3058] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:38.972533 29738 leader_election.cc:290] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:38.978582 28001 raft_consensus.cc:2466] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:38.987260 28125 leader_election.cc:304] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:38.988085 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:38.991515 28154 raft_consensus.cc:2466] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
W20250905 19:59:38.994062 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:38.962174 29763 raft_consensus.cc:3058] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:38.994801 27975 leader_election.cc:304] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:38.997112 29511 raft_consensus.cc:491] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
W20250905 19:59:39.003682 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 47433.1us
W20250905 19:59:39.018038 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 67134.3us
I20250905 19:59:39.021937 28122 leader_election.cc:304] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:39.025884 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:39.026753 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:39.026670 29644 consensus_queue.cc:1035] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 19:59:38.998919 29742 raft_consensus.cc:2747] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:39.027595 28082 raft_consensus.cc:2391] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 2: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:39.003548 29645 raft_consensus.cc:2802] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:38.999814 28076 raft_consensus.cc:2466] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:39.041114 29626 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:39.001855 29737 raft_consensus.cc:2802] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:39.038990 29805 raft_consensus.cc:2802] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election won for term 3
W20250905 19:59:39.064898 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 59518.7us
I20250905 19:59:39.000521 29511 raft_consensus.cc:513] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:39.036435 29815 raft_consensus.cc:2747] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 19:59:39.049503 29644 raft_consensus.cc:491] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:39.077389 29644 raft_consensus.cc:513] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:39.061550 29800 consensus_queue.cc:1035] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20250905 19:59:39.054965 29737 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:39.033429 29714 consensus_queue.cc:1046] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
W20250905 19:59:39.084012 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 84396.4us
W20250905 19:59:39.100775 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 106062us
I20250905 19:59:39.079608 29644 leader_election.cc:290] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:39.103991 28122 leader_election.cc:304] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:39.065927 29645 raft_consensus.cc:695] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:39.106616 28085 raft_consensus.cc:3058] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:39.072794 29813 raft_consensus.cc:3053] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:39.082271 29508 raft_consensus.cc:491] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:39.124466 28156 raft_consensus.cc:1273] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:39.083786 29737 raft_consensus.cc:3058] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:39.120707 28122 leader_election.cc:304] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:39.146399 28001 raft_consensus.cc:1273] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 19:59:39.102916 29805 raft_consensus.cc:695] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:39.109277 29757 raft_consensus.cc:2747] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 19:59:39.107292 29756 raft_consensus.cc:491] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:39.112543 29622 raft_consensus.cc:491] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:39.112335 29644 raft_consensus.cc:491] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:39.111536 29813 raft_consensus.cc:738] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:39.155509 29758 raft_consensus.cc:2802] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:39.174937 28156 raft_consensus.cc:1273] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:39.172776 28122 leader_election.cc:304] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:39.154922 29508 raft_consensus.cc:513] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:39.201160 29788 raft_consensus.cc:2802] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:39.202797 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:39.203157 29788 raft_consensus.cc:695] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:39.203485 29511 leader_election.cc:290] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:39.174141 29756 raft_consensus.cc:513] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:39.204249 29804 raft_consensus.cc:3053] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Stepping down as leader of term 4
I20250905 19:59:39.204710 29804 raft_consensus.cc:738] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:39.199779 29758 raft_consensus.cc:695] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:39.207532 29756 leader_election.cc:290] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:39.197263 29645 consensus_queue.cc:237] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:39.110312 29810 consensus_queue.cc:1046] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.070s
I20250905 19:59:39.203490 28747 raft_consensus.cc:2466] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
I20250905 19:59:39.204134 29788 consensus_queue.cc:237] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:39.214655 29505 raft_consensus.cc:3053] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:39.215124 29505 raft_consensus.cc:738] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:39.216040 29505 consensus_queue.cc:260] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:39.217550 29505 raft_consensus.cc:3058] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
W20250905 19:59:39.220189 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:39.205991 29758 consensus_queue.cc:237] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 19:59:39.113480 28157 raft_consensus.cc:1273] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:39.183045 28085 raft_consensus.cc:1273] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250905 19:59:39.183166 29816 raft_consensus.cc:491] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:39.247565 28049 leader_election.cc:304] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:39.205693 29804 consensus_queue.cc:260] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:39.268122 29823 raft_consensus.cc:2802] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:39.268817 29823 raft_consensus.cc:491] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:39.269302 29823 raft_consensus.cc:3058] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:39.256492 29826 consensus_queue.cc:1035] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:39.196571 29745 consensus_queue.cc:1035] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.069s
I20250905 19:59:39.247102 29800 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:39.195892 29644 raft_consensus.cc:513] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:39.196949 29622 raft_consensus.cc:513] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:39.247437 29822 raft_consensus.cc:491] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:39.264423 29599 raft_consensus.cc:491] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:39.276129 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:39.305483 28003 raft_consensus.cc:1273] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250905 19:59:39.288368 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:59:39.305305 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 57327.8us
I20250905 19:59:39.310693 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:39.286672 29626 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:39.312655 28001 raft_consensus.cc:1273] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:39.310489 29599 raft_consensus.cc:513] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:39.308645 29804 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:39.352144 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:39.355248 28155 raft_consensus.cc:1273] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250905 19:59:39.355793 28749 raft_consensus.cc:3058] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:39.290343 29816 raft_consensus.cc:513] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:39.363850 29816 leader_election.cc:290] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:39.365172 29823 raft_consensus.cc:513] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:39.366560 29599 leader_election.cc:290] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:39.254333 29805 consensus_queue.cc:237] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:39.368948 28005 raft_consensus.cc:2466] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:59:39.344938 28076 raft_consensus.cc:3058] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:39.301998 29800 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:39.381434 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:39.383760 28748 raft_consensus.cc:2391] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 2: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 19:59:39.353021 29626 consensus_queue.cc:1035] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
W20250905 19:59:39.396553 29749 consensus_meta.cc:287] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.671s user 0.005s sys 0.007s
I20250905 19:59:39.396855 29813 consensus_queue.cc:260] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:39.398473 29644 leader_election.cc:290] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:39.341777 29822 raft_consensus.cc:513] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:39.400177 27905 catalog_manager.cc:5582] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 2 to 3, leader changed from 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) to 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132). New cstate: current_term: 3 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250905 19:59:39.400950 27894 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet e707b0f561934674947fb463ce626a07 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 26b2f874b8e54f08b01a9ed00d5cce05 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:39.401554 29822 leader_election.cc:290] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:39.405258 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:39.405525 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:39.405978 28150 raft_consensus.cc:2466] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:39.406086 28159 raft_consensus.cc:2466] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:39.407537 28050 leader_election.cc:304] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:39.407684 29763 raft_consensus.cc:513] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:39.409761 28050 leader_election.cc:304] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:39.411027 29738 raft_consensus.cc:491] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:39.411940 29822 raft_consensus.cc:2802] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:39.413679 29800 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:39.413308 27905 catalog_manager.cc:5582] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 3 to 4, leader changed from 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 4 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:39.414062 29711 raft_consensus.cc:2802] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:39.419651 29711 raft_consensus.cc:491] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:39.420089 29711 raft_consensus.cc:3058] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:39.383435 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:39.421185 28752 raft_consensus.cc:2391] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 2: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 19:59:39.422168 29619 raft_consensus.cc:491] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:39.422811 29619 raft_consensus.cc:513] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:39.424201 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:39.425006 28157 raft_consensus.cc:3058] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:39.425096 29619 leader_election.cc:290] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:39.381039 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:39.350651 29713 consensus_queue.cc:1035] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20250905 19:59:39.427558 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:39.417688 29811 log.cc:927] Time spent T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.239s user 0.003s sys 0.000s
I20250905 19:59:39.428359 28079 raft_consensus.cc:2391] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 3: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:39.411636 29738 raft_consensus.cc:513] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:39.411981 29622 leader_election.cc:290] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:39.432585 28077 raft_consensus.cc:1273] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250905 19:59:39.434217 27904 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34448 (request call id 252) took 1151 ms. Trace:
I20250905 19:59:39.418529 29812 raft_consensus.cc:3053] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Stepping down as leader of term 4
I20250905 19:59:39.436265 29812 raft_consensus.cc:738] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:39.418154 29822 raft_consensus.cc:491] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:39.432191 29737 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:39.457705 28749 raft_consensus.cc:2466] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
I20250905 19:59:39.487406 28049 leader_election.cc:304] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:39.487905 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:59:39.432073 29787 raft_consensus.cc:513] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:39.502750 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:39.398422 29813 raft_consensus.cc:3058] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:39.501677 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:39.437403 29812 consensus_queue.cc:260] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:39.524329 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:39.538517 28077 raft_consensus.cc:2466] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 3.
I20250905 19:59:39.540151 28080 raft_consensus.cc:1273] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:39.540365 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:39.541110 28082 raft_consensus.cc:2366] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 2. Current term is 3.
I20250905 19:59:39.543545 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:39.544201 28084 raft_consensus.cc:3058] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:39.529597 28155 raft_consensus.cc:2466] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
I20250905 19:59:39.463376 29822 raft_consensus.cc:3058] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:39.454041 29763 leader_election.cc:290] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:39.535573 28079 raft_consensus.cc:1273] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:39.545920 28715 leader_election.cc:304] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:39.486954 29627 raft_consensus.cc:491] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:39.550719 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:39.491997 29823 leader_election.cc:290] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:39.558480 28078 raft_consensus.cc:1273] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:39.560876 28076 raft_consensus.cc:1273] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:59:39.572690 29738 leader_election.cc:290] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:39.551995 29511 raft_consensus.cc:491] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:39.576335 29511 raft_consensus.cc:513] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:39.557647 29668 raft_consensus.cc:491] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:39.578022 29668 raft_consensus.cc:513] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:39.559085 29824 consensus_queue.cc:1035] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20250905 19:59:39.580739 29668 leader_election.cc:290] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:39.545374 29622 raft_consensus.cc:2747] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:39.582080 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:39.582937 28083 raft_consensus.cc:2391] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 3: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:39.582849 29812 raft_consensus.cc:3058] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:39.584574 29833 raft_consensus.cc:491] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:39.585136 29833 raft_consensus.cc:513] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:39.587422 29833 leader_election.cc:290] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:39.575606 29801 raft_consensus.cc:2802] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:39.571880 28081 raft_consensus.cc:2466] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:39.434739 27904 rpcz_store.cc:276] 0905 19:59:38.283146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:38.283561 (+ 415us) service_pool.cc:225] Handling call
0905 19:59:39.434152 (+1150591us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:59:38.722586 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:59:38.723187 (+ 601us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:59:38.723236 (+ 49us) write_op.cc:435] Acquired schema lock
0905 19:59:38.723262 (+ 26us) tablet.cc:660] Decoding operations
0905 19:59:38.723603 (+ 341us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:59:38.723669 (+ 66us) write_op.cc:641] Partition lock acquired for write op
0905 19:59:38.723694 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:59:38.723893 (+ 199us) tablet.cc:699] Row locks acquired
0905 19:59:38.723910 (+ 17us) write_op.cc:260] PREPARE: finished
0905 19:59:38.724052 (+ 142us) write_op.cc:270] Start()
0905 19:59:38.724164 (+ 112us) write_op.cc:276] Timestamp: P: 1757102378724007 usec, L: 0
0905 19:59:38.724182 (+ 18us) op_driver.cc:348] REPLICATION: starting
0905 19:59:38.812277 (+ 88095us) log.cc:844] Serialized 536 byte log entry
0905 19:59:38.990323 (+178046us) op_driver.cc:464] REPLICATION: finished
0905 19:59:38.990758 (+ 435us) write_op.cc:301] APPLY: starting
0905 19:59:38.990896 (+ 138us) tablet.cc:1365] starting BulkCheckPresence
0905 19:59:39.003161 (+ 12265us) tablet.cc:1368] finished BulkCheckPresence
0905 19:59:39.015758 (+ 12597us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:59:39.042545 (+ 26787us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:59:39.042764 (+ 219us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:59:39.042829 (+ 65us) write_op.cc:312] APPLY: finished
0905 19:59:39.043097 (+ 268us) log.cc:844] Serialized 23 byte log entry
0905 19:59:39.043335 (+ 238us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:59:39.043442 (+ 107us) write_op.cc:454] Released schema lock
0905 19:59:39.043503 (+ 61us) write_op.cc:341] FINISH: Updating metrics
0905 19:59:39.044026 (+ 523us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:59:39.044077 (+ 51us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":20099,"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":196,"apply.run_cpu_time_us":2375,"apply.run_wall_time_us":53161,"num_ops":1,"prepare.queue_time_us":122,"prepare.run_cpu_time_us":2994,"prepare.run_wall_time_us":118032,"replication_time_us":265976}]]}
I20250905 19:59:39.589581 28157 raft_consensus.cc:2466] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
W20250905 19:59:39.590065 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 57045us
I20250905 19:59:39.591727 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:39.595764 27973 leader_election.cc:400] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Vote denied by peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193) with higher term. Message: Invalid argument: T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 2. Current term is 3.
W20250905 19:59:39.592540 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 47420.4us
I20250905 19:59:39.596138 27973 leader_election.cc:403] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20250905 19:59:39.596676 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 44404.9us
I20250905 19:59:39.597425 28050 leader_election.cc:304] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:39.594334 28049 leader_election.cc:304] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:39.546118 29815 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:39.601622 28156 raft_consensus.cc:2466] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
I20250905 19:59:39.613276 28050 leader_election.cc:304] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:39.618877 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:39.620766 28155 raft_consensus.cc:1273] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:59:39.625126 28084 raft_consensus.cc:2466] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:59:39.625910 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 19:59:39.631150 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 82506.8us
I20250905 19:59:39.550585 29508 leader_election.cc:290] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:39.636633 28005 raft_consensus.cc:1273] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:59:39.636021 29644 raft_consensus.cc:2953] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:39.622205 28009 raft_consensus.cc:1238] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 4. Current term is 5. Ops: []
I20250905 19:59:39.638993 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:39.639643 28083 raft_consensus.cc:2466] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:39.591951 29627 raft_consensus.cc:513] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:39.607419 29668 raft_consensus.cc:3058] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 3
I20250905 19:59:39.643020 29627 leader_election.cc:290] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:39.643150 29805 raft_consensus.cc:491] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:39.644127 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:39.643926 29805 raft_consensus.cc:513] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:39.644846 28001 raft_consensus.cc:2466] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:39.645816 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:39.646565 28008 raft_consensus.cc:2466] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 19:59:39.648022 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:39.649099 29622 raft_consensus.cc:2802] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:39.649843 29622 raft_consensus.cc:491] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:39.648967 28748 raft_consensus.cc:2466] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:39.651136 29622 raft_consensus.cc:3058] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:39.652261 28123 leader_election.cc:304] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 19:59:39.550701 29599 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:39.654749 29809 raft_consensus.cc:2802] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:39.655141 29809 raft_consensus.cc:491] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:39.655493 29809 raft_consensus.cc:3058] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:39.653918 29599 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:39.605566 29768 raft_consensus.cc:2802] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:39.657471 28003 raft_consensus.cc:2466] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:39.657604 29768 raft_consensus.cc:695] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 4 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:39.657678 29599 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:39.658635 29768 consensus_queue.cc:237] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:39.650406 29805 leader_election.cc:290] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 19:59:39.663796 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:39.606524 29711 raft_consensus.cc:513] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:39.668217 29711 leader_election.cc:290] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:39.564334 29814 consensus_queue.cc:1035] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.183s
I20250905 19:59:39.595959 29788 consensus_queue.cc:1035] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.075s
I20250905 19:59:39.671563 27973 leader_election.cc:304] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:39.672963 27973 leader_election.cc:304] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:39.673785 27973 leader_election.cc:304] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:39.568280 29737 leader_election.cc:290] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:39.567060 29787 leader_election.cc:290] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 19:59:39.676681 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 130853us
I20250905 19:59:39.533334 29826 consensus_queue.cc:1035] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.075s
I20250905 19:59:39.592933 29801 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250905 19:59:39.680938 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 47779.8us
I20250905 19:59:39.607723 29833 raft_consensus.cc:2802] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:39.684813 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:39.701704 28009 raft_consensus.cc:3058] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:39.701926 28125 leader_election.cc:304] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:39.674661 29756 raft_consensus.cc:491] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:39.707783 28125 leader_election.cc:304] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:39.708926 28125 leader_election.cc:304] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:39.709492 28003 raft_consensus.cc:2466] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
W20250905 19:59:39.710279 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 397903us
I20250905 19:59:39.677062 29711 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:39.680022 29713 raft_consensus.cc:2802] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:39.733120 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:39.735226 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:39.680258 29755 raft_consensus.cc:2802] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:39.691592 29801 raft_consensus.cc:3058] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:39.751399 28010 raft_consensus.cc:2466] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:59:39.702749 29738 raft_consensus.cc:491] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:39.681653 29834 raft_consensus.cc:2802] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:39.767329 29814 raft_consensus.cc:491] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:39.730327 29619 consensus_queue.cc:1035] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
I20250905 19:59:39.707546 29668 raft_consensus.cc:2747] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193) with higher term. Message: Invalid argument: T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 2. Current term is 3.
I20250905 19:59:39.678598 29816 consensus_queue.cc:1035] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.062s
I20250905 19:59:39.704839 29833 raft_consensus.cc:491] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:39.738166 29805 raft_consensus.cc:2802] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:39.773021 28009 raft_consensus.cc:1273] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:39.674904 29815 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:39.822512 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:39.770844 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:39.774784 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:39.768213 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:39.800318 28010 raft_consensus.cc:1273] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250905 19:59:39.774227 28006 raft_consensus.cc:1273] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:59:39.773523 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:39.931576 28001 raft_consensus.cc:2391] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:39.679649 29763 consensus_queue.cc:1035] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.084s
I20250905 19:59:39.820230 29834 raft_consensus.cc:491] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:39.747763 29711 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:39.748595 29713 raft_consensus.cc:491] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:39.830217 29814 raft_consensus.cc:513] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:39.787125 28745 raft_consensus.cc:2953] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:39.939018 29834 raft_consensus.cc:3058] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:39.966045 29713 raft_consensus.cc:3058] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:39.945185 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:39.929255 29824 raft_consensus.cc:491] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:39.971311 29824 raft_consensus.cc:513] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:39.772035 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:39.790645 29737 raft_consensus.cc:491] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:39.972344 28744 raft_consensus.cc:3058] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:39.972545 29737 raft_consensus.cc:513] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:39.956288 29745 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:39.974092 29745 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:39.975666 29737 leader_election.cc:290] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:39.748135 29511 leader_election.cc:290] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:39.758400 29757 raft_consensus.cc:491] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:39.976580 29745 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:39.976902 29757 raft_consensus.cc:513] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:39.961824 29822 raft_consensus.cc:513] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:39.979530 29757 leader_election.cc:290] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:39.980080 29814 leader_election.cc:290] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:39.753959 29804 consensus_queue.cc:1035] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.228s
I20250905 19:59:39.981918 29824 leader_election.cc:290] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:39.910250 29788 raft_consensus.cc:491] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:39.791252 29749 raft_consensus.cc:2802] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:39.771812 28008 raft_consensus.cc:3058] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:39.831943 29833 raft_consensus.cc:3058] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:39.792146 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:39.762647 29787 raft_consensus.cc:2802] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 19:59:39.723987 29756 raft_consensus.cc:513] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:39.747242 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:39.991488 29826 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.038410 28748 raft_consensus.cc:3058] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:40.032325 29787 raft_consensus.cc:491] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:40.051746 29756 leader_election.cc:290] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:40.052184 29787 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:40.052498 29645 raft_consensus.cc:491] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:40.053069 29645 raft_consensus.cc:513] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.055898 29713 raft_consensus.cc:513] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:40.058106 29713 leader_election.cc:290] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:39.930132 28002 raft_consensus.cc:2391] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 2: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:39.935825 28009 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582"
dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
mode: GRACEFUL
new_leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
from {username='slave'} at 127.0.0.1:47564
I20250905 19:59:39.772233 29738 raft_consensus.cc:513] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:39.961735 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:39.830677 29805 raft_consensus.cc:491] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:39.787289 29755 raft_consensus.cc:695] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:39.988248 29511 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:39.990146 29508 raft_consensus.cc:491] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:39.990852 29832 raft_consensus.cc:491] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.104830 29832 raft_consensus.cc:513] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.106863 29832 leader_election.cc:290] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:40.107427 29832 consensus_queue.cc:1035] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:39.992323 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:40.017534 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:40.109658 28157 raft_consensus.cc:3058] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:40.110150 29755 consensus_queue.cc:237] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:40.113621 29508 raft_consensus.cc:513] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.062052 29826 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.115906 29508 leader_election.cc:290] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:40.116470 29826 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:40.118000 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:40.019034 28152 raft_consensus.cc:1273] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:39.929855 28005 raft_consensus.cc:2391] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:40.035295 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:40.120095 28001 raft_consensus.cc:2391] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 2: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:39.958971 29815 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:39.979558 29822 leader_election.cc:290] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:40.051774 28748 raft_consensus.cc:2466] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:40.059334 29505 raft_consensus.cc:491] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:39.965739 29622 raft_consensus.cc:513] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.060209 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:40.003262 29823 raft_consensus.cc:491] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:40.010738 29599 raft_consensus.cc:491] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.016678 29627 raft_consensus.cc:491] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.082360 28009 raft_consensus.cc:604] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Received request to transfer leadership to TS e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:40.160004 28009 raft_consensus.cc:612] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250905 19:59:39.927757 28749 raft_consensus.cc:2466] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
I20250905 19:59:40.034610 29749 raft_consensus.cc:695] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:39.910185 29813 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:40.084929 27974 leader_election.cc:304] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:40.096127 29800 consensus_queue.cc:1035] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20250905 19:59:40.098783 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:40.101444 29757 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.102624 29511 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.121014 28157 raft_consensus.cc:2466] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 19:59:39.936004 29668 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.124783 28003 raft_consensus.cc:2466] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:40.126793 28751 raft_consensus.cc:2391] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 2: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 19:59:40.130227 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:40.132652 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:40.015671 29788 raft_consensus.cc:513] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:40.173328 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:40.171923 28002 raft_consensus.cc:2466] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 3.
I20250905 19:59:40.091456 29805 raft_consensus.cc:3058] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:40.090449 29768 consensus_queue.cc:1035] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.072s
I20250905 19:59:40.091723 29714 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
W20250905 19:59:40.167091 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 82265.9us
I20250905 19:59:40.133237 29619 raft_consensus.cc:491] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.179000 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:40.175956 29668 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.179047 29619 raft_consensus.cc:513] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.181563 29668 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:40.173064 28747 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 19:59:40.175489 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:40.174881 28718 leader_election.cc:304] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:40.151419 28744 raft_consensus.cc:2466] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:59:40.184374 29840 raft_consensus.cc:2802] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:40.184758 29840 raft_consensus.cc:491] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:40.185088 29840 raft_consensus.cc:3058] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:40.179176 29745 raft_consensus.cc:491] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.185798 29745 raft_consensus.cc:513] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.186691 28158 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "6db33851e9914ca58560e8902701dca3"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
from {username='slave'} at 127.0.0.1:36164
I20250905 19:59:40.187047 29511 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:40.186820 29749 consensus_queue.cc:237] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.187362 28158 raft_consensus.cc:604] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Received request to transfer leadership to TS e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:40.165671 29823 raft_consensus.cc:513] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.175684 29850 raft_consensus.cc:2747] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:40.178725 29833 raft_consensus.cc:513] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.164227 29711 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:40.189302 27974 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:40.187912 28158 raft_consensus.cc:612] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250905 19:59:40.151880 29505 raft_consensus.cc:513] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.194106 28155 raft_consensus.cc:2466] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:59:40.174628 29645 leader_election.cc:290] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:40.164577 29627 raft_consensus.cc:513] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:40.195582 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 203873us
I20250905 19:59:40.165246 29599 raft_consensus.cc:513] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.188050 29745 leader_election.cc:290] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:40.228766 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:40.171105 29813 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:40.219247 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:40.229856 28153 raft_consensus.cc:2466] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
I20250905 19:59:40.239233 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:40.202347 29713 raft_consensus.cc:2802] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 19:59:40.190187 29737 raft_consensus.cc:491] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.236279 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:40.237185 28154 raft_consensus.cc:2466] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:40.240252 28157 raft_consensus.cc:2466] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
I20250905 19:59:40.241080 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 19:59:40.195308 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 204024us
W20250905 19:59:40.242082 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 209530us
I20250905 19:59:40.204900 29619 leader_election.cc:290] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:40.173835 29757 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:40.261273 27975 leader_election.cc:304] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:40.210796 29815 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.266839 28049 leader_election.cc:304] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:40.268494 28151 raft_consensus.cc:2466] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:40.221190 29645 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.274298 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:40.236097 29810 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.274333 28752 raft_consensus.cc:3058] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Advancing to term 4
I20250905 19:59:40.285434 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:40.289621 28151 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
I20250905 19:59:40.290588 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:40.236150 29711 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:40.295807 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:40.251953 29713 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:40.296761 28083 raft_consensus.cc:2466] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 3.
W20250905 19:59:40.331135 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 69983.7us
I20250905 19:59:40.196270 29714 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:40.345528 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 78839.3us
I20250905 19:59:40.356491 28154 raft_consensus.cc:1273] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250905 19:59:40.358168 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:40.359254 28009 raft_consensus.cc:2466] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:40.252319 29787 raft_consensus.cc:513] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.349982 28005 raft_consensus.cc:1273] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:40.349481 28049 leader_election.cc:304] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:40.272672 29822 consensus_queue.cc:1035] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.088s
I20250905 19:59:40.350271 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:40.369506 28001 raft_consensus.cc:2466] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:40.340843 29713 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 19:59:40.276575 29841 consensus_queue.cc:1035] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.084s
I20250905 19:59:40.288383 29815 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:40.358927 29809 consensus_meta.cc:287] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.703s user 0.010s sys 0.000s
I20250905 19:59:40.346174 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:40.345832 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:40.264113 29737 raft_consensus.cc:513] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.292960 29619 consensus_queue.cc:1035] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20250905 19:59:40.390504 28084 raft_consensus.cc:1273] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:40.349233 29831 raft_consensus.cc:2802] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:40.294023 29645 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.379742 29622 leader_election.cc:290] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:40.405093 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 65047.4us
I20250905 19:59:40.354049 29738 leader_election.cc:290] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:40.410221 28150 raft_consensus.cc:1238] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20250905 19:59:40.363792 29755 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.333329 29810 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:40.411522 29668 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.367319 29818 raft_consensus.cc:491] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.340515 29852 consensus_queue.cc:1035] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.101s
I20250905 19:59:40.424683 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:40.355567 29711 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.365175 29511 raft_consensus.cc:491] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.447616 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:40.461259 28003 raft_consensus.cc:2391] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:40.461696 28009 raft_consensus.cc:2466] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
I20250905 19:59:40.404837 29856 raft_consensus.cc:491] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.451165 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:40.456796 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:40.459810 28154 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:36164
I20250905 19:59:40.459982 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:40.433323 28008 raft_consensus.cc:1273] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:40.472877 28084 raft_consensus.cc:1273] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:40.481168 28154 raft_consensus.cc:604] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Received request to transfer leadership to TS cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:40.524694 28154 raft_consensus.cc:612] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250905 19:59:40.525579 28005 raft_consensus.cc:2391] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:40.420425 28748 raft_consensus.cc:2466] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:40.476315 29852 raft_consensus.cc:491] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.531093 29854 raft_consensus.cc:491] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.531244 29849 raft_consensus.cc:491] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:40.531733 29854 raft_consensus.cc:513] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.531852 29849 raft_consensus.cc:513] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:40.476529 29505 leader_election.cc:290] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:40.474754 29818 raft_consensus.cc:513] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:40.534941 29857 log.cc:927] Time spent T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.113s user 0.000s sys 0.001s
I20250905 19:59:40.536067 28001 raft_consensus.cc:2391] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:40.474560 29668 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:40.530912 29852 raft_consensus.cc:513] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.448711 29738 raft_consensus.cc:491] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.449110 29833 leader_election.cc:290] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:40.414644 29809 raft_consensus.cc:513] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:40.453624 29644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:59536 (request call id 526) took 1754 ms. Trace:
W20250905 19:59:40.473737 29845 log.cc:927] Time spent T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.251s user 0.000s sys 0.001s
I20250905 19:59:40.478274 29823 leader_election.cc:290] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:40.456905 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:40.581400 29862 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.447397 29831 raft_consensus.cc:695] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:40.582056 28076 raft_consensus.cc:2466] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:40.582763 29831 consensus_queue.cc:237] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:40.540234 29627 leader_election.cc:290] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:40.460052 29864 raft_consensus.cc:2802] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:40.585662 29864 raft_consensus.cc:695] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:40.529525 29856 raft_consensus.cc:513] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.586644 29864 consensus_queue.cc:237] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:40.587234 29865 raft_consensus.cc:491] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.587775 29865 raft_consensus.cc:513] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.472718 29847 raft_consensus.cc:491] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.589627 29847 raft_consensus.cc:513] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:40.590745 29756 raft_consensus.cc:491] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:40.455919 29804 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:40.591284 29756 raft_consensus.cc:513] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:40.591398 29804 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:40.473451 29855 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:40.529872 29511 raft_consensus.cc:513] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.593025 29855 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:40.442306 29834 raft_consensus.cc:513] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:40.594864 28007 raft_consensus.cc:2466] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
I20250905 19:59:40.459954 29755 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:40.596293 29853 raft_consensus.cc:2802] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:40.597117 29834 leader_election.cc:290] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:40.600310 29853 raft_consensus.cc:491] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:40.600808 29853 raft_consensus.cc:3058] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:40.553140 29738 raft_consensus.cc:513] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.554025 29848 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
W20250905 19:59:40.547835 27894 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet e707b0f561934674947fb463ce626a07 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 26b2f874b8e54f08b01a9ed00d5cce05 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:40.565133 28125 leader_election.cc:304] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 19:59:40.615607 27894 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 67835.1us
I20250905 19:59:40.536511 28152 raft_consensus.cc:2466] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 3.
I20250905 19:59:40.540647 29816 raft_consensus.cc:491] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.539564 29850 raft_consensus.cc:491] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.556367 29876 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:40.616034 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 50993.8us
I20250905 19:59:40.638626 28125 leader_election.cc:304] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:40.639393 28752 raft_consensus.cc:2466] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 19:59:40.641521 29854 leader_election.cc:290] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:40.642499 29711 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:40.641606 29876 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.641558 29757 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:40.644320 29755 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:40.643004 29852 leader_election.cc:290] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:40.641423 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:40.649611 29505 consensus_queue.cc:1035] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20250905 19:59:40.641431 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:40.652007 28076 raft_consensus.cc:2466] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:40.652160 29804 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:40.652367 28745 raft_consensus.cc:3058] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:40.616654 29813 leader_election.cc:290] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:40.652781 29737 leader_election.cc:290] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:40.652942 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:40.653591 28748 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:40.654707 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:40.582085 29862 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:40.648089 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:40.656390 28159 raft_consensus.cc:2366] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a for earlier term 3. Current term is 4.
I20250905 19:59:40.649228 28084 raft_consensus.cc:1273] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250905 19:59:40.657505 29862 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:40.637532 29809 leader_election.cc:290] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:40.658556 29619 raft_consensus.cc:2802] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:40.658977 29619 raft_consensus.cc:491] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:40.658862 29668 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:40.648629 29815 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:40.659616 29850 raft_consensus.cc:513] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 19:59:40.660431 29866 raft_consensus.cc:2802] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:40.660045 29813 raft_consensus.cc:491] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.566275 29644 rpcz_store.cc:276] 0905 19:59:38.699062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:38.699187 (+ 125us) service_pool.cc:225] Handling call
0905 19:59:38.753379 (+ 54192us) log.cc:844] Serialized 501 byte log entry
0905 19:59:38.991566 (+238187us) spinlock_profiling.cc:243] Waited 4.63 ms on lock 0x7b18003f3640. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49ac19f4a 00007fe4a4107409 00007fe49aadb3e0 00007fe49aca14e4 00007fe49aca1455 00007fe49db922db 00007fe49dbf7bd3 00007fe49dc1f702 00007fe49dc179ba
0905 19:59:39.106750 (+115184us) spinlock_profiling.cc:243] Waited 7.21 ms on lock 0x7b18003f3640. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49ac723d6 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49bfb1980 00007fe49bfc27b2 00007fe49bfc2d12 00007fe49bf877cb
0905 19:59:39.413777 (+307027us) spinlock_profiling.cc:243] Waited 2.65 ms on lock 0x7b4c002c8e80. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49bfb1980 00007fe49bfc27b2 00007fe49bfc2d12 00007fe49bf877cb 00007fe49bfb4e9c 00007fe49bfb535e 00007fe49c221ceb 00007fe49db94251 00007fe49db92859 00007fe49db97911 00007fe49db97899
0905 19:59:40.452085 (+1038308us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":84924,"raft.queue_time_us":213176,"raft.run_cpu_time_us":15864,"raft.run_wall_time_us":492287,"spinlock_wait_cycles":33322240,"thread_start_us":29540,"threads_started":1,"wal-append.queue_time_us":1651,"wal-append.run_cpu_time_us":5831,"wal-append.run_wall_time_us":1136970}
I20250905 19:59:40.660854 29866 raft_consensus.cc:491] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:40.661324 29866 raft_consensus.cc:3058] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:40.661104 29813 raft_consensus.cc:513] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:40.662895 28717 leader_election.cc:400] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 pre-election: Vote denied by peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705) with higher term. Message: Invalid argument: T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a for earlier term 3. Current term is 4.
I20250905 19:59:40.662966 29847 leader_election.cc:290] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:40.658880 29810 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:40.646123 29816 raft_consensus.cc:513] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.665120 29757 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.665668 29757 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.666018 29816 leader_election.cc:290] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:40.663333 28717 leader_election.cc:403] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20250905 19:59:40.667840 29757 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:40.659459 29645 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:40.676435 29865 leader_election.cc:290] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:40.677567 29855 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:40.669153 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:40.659489 29619 raft_consensus.cc:3058] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:40.669584 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:40.678407 28748 raft_consensus.cc:2466] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:40.678634 28159 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 2.
I20250905 19:59:40.663251 29813 leader_election.cc:290] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:40.654778 29787 leader_election.cc:290] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:40.638253 29848 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:40.684262 28156 raft_consensus.cc:2466] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 2.
I20250905 19:59:40.685667 28123 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 19:59:40.683833 29848 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 19:59:40.689633 29801 consensus_meta.cc:287] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.901s user 0.005s sys 0.006s
I20250905 19:59:40.691735 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:40.657779 29840 raft_consensus.cc:513] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:40.700661 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:40.645028 29876 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:40.684989 29815 raft_consensus.cc:2747] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705) with higher term. Message: Invalid argument: T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a for earlier term 3. Current term is 4.
I20250905 19:59:40.684620 29599 leader_election.cc:290] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:40.669836 29856 leader_election.cc:290] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:40.699530 29787 raft_consensus.cc:2802] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:40.708418 29511 leader_election.cc:290] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:40.723054 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:40.724231 28084 raft_consensus.cc:2391] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 2: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:40.712714 28748 raft_consensus.cc:2466] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:40.707521 29813 raft_consensus.cc:491] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:40.732209 28715 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:40.709264 29756 leader_election.cc:290] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:40.737330 28078 raft_consensus.cc:1273] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:59:40.743422 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:40.745460 28084 raft_consensus.cc:2466] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:40.745815 28123 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 19:59:40.709456 29848 raft_consensus.cc:491] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:40.720888 29855 raft_consensus.cc:491] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.759414 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:40.723376 29787 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:40.777927 29869 raft_consensus.cc:491] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:40.726244 29849 leader_election.cc:290] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:40.697877 29801 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:40.778908 29826 raft_consensus.cc:491] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:40.746722 29815 raft_consensus.cc:2802] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:40.748914 29818 leader_election.cc:290] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:40.760042 29738 leader_election.cc:290] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:40.793562 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:40.803517 28078 raft_consensus.cc:2391] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 2: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:40.780400 29787 raft_consensus.cc:3058] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:40.807582 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:40.808475 28749 raft_consensus.cc:2466] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:59:40.779523 29804 raft_consensus.cc:491] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.832315 27974 leader_election.cc:304] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:40.778576 29869 raft_consensus.cc:513] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:40.793488 29756 raft_consensus.cc:491] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.748703 29813 raft_consensus.cc:513] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:40.830756 29508 raft_consensus.cc:491] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.794657 29815 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:40.831851 29800 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.857023 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:40.779527 29855 raft_consensus.cc:513] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 19:59:40.849632 28122 leader_election.cc:304] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:40.864265 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:40.779358 29848 raft_consensus.cc:513] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:40.844211 29644 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.847808 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:40.844491 29876 raft_consensus.cc:2802] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:40.885340 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:59:40.902124 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 52632.4us
I20250905 19:59:40.856660 28747 raft_consensus.cc:1238] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:40.909552 28122 leader_election.cc:304] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:40.807463 29826 raft_consensus.cc:513] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.903393 28159 raft_consensus.cc:2466] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
W20250905 19:59:40.904935 28155 consensus_meta.cc:287] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.650s user 0.015s sys 0.005s
I20250905 19:59:40.853869 29871 raft_consensus.cc:491] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.860141 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:40.915902 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:40.885561 29873 raft_consensus.cc:2802] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:40.920173 29855 leader_election.cc:290] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421)
I20250905 19:59:40.908465 29876 raft_consensus.cc:491] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:40.920367 29619 raft_consensus.cc:513] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.853256 29757 consensus_queue.cc:1035] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.062s
I20250905 19:59:40.869745 29508 raft_consensus.cc:513] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.898552 29882 consensus_queue.cc:1035] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20250905 19:59:40.870795 29810 raft_consensus.cc:491] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.921556 29862 raft_consensus.cc:491] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.900378 29854 raft_consensus.cc:491] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:40.922552 29867 raft_consensus.cc:491] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:40.923527 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:40.923913 29850 leader_election.cc:290] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:40.913409 29809 raft_consensus.cc:2802] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:40.842931 29804 raft_consensus.cc:513] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:40.913928 29644 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:40.926523 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:40.928010 28155 raft_consensus.cc:2466] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:40.868835 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:40.890980 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:40.890846 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:40.916450 29505 raft_consensus.cc:2802] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:40.941562 29826 leader_election.cc:290] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:40.942962 28752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40904 (request call id 470) took 1220 ms. Trace:
I20250905 19:59:40.945354 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:40.947712 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:40.951371 29866 raft_consensus.cc:513] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:40.960233 28007 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1833) took 1218 ms (client timeout 1531 ms). Trace:
I20250905 19:59:40.927372 29849 consensus_queue.cc:1035] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20250905 19:59:40.970887 28744 raft_consensus.cc:2466] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:59:40.963687 29848 leader_election.cc:290] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:40.962656 29840 leader_election.cc:290] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:41.007808 29619 leader_election.cc:290] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:41.002395 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:40.972429 29873 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:40.981429 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:41.043721 28078 raft_consensus.cc:2466] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:41.044142 29873 raft_consensus.cc:3058] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:41.044292 28751 raft_consensus.cc:2466] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:40.985880 29599 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:41.053473 29599 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:41.030028 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:40.991595 29854 raft_consensus.cc:513] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:41.010632 28748 raft_consensus.cc:2391] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 2: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 19:59:40.994562 28080 raft_consensus.cc:2466] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:40.890530 29756 raft_consensus.cc:513] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:41.032624 29813 leader_election.cc:290] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:40.972201 29869 leader_election.cc:290] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:41.000304 27904 catalog_manager.cc:5582] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 1 to 2, leader changed from e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 2 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:41.036194 29714 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:41.068603 27975 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.614s user 0.015s sys 0.064s
I20250905 19:59:40.995441 27905 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:60818 (request call id 146) took 2367 ms. Trace:
I20250905 19:59:41.034184 29801 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:41.012890 28745 raft_consensus.cc:2466] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:41.035486 29508 leader_election.cc:290] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:41.005172 29853 raft_consensus.cc:513] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:41.044999 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:41.074708 27904 catalog_manager.cc:5582] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 1 to 2, leader changed from 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 2 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:41.075929 28078 raft_consensus.cc:2466] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 19:59:41.045799 28123 leader_election.cc:304] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 19:59:40.871122 29815 raft_consensus.cc:3058] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:40.992760 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:41.042300 28153 raft_consensus.cc:2466] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
W20250905 19:59:41.036185 28007 rpcz_store.cc:269] 0905 19:59:39.741522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:39.741795 (+ 273us) service_pool.cc:225] Handling call
0905 19:59:40.960017 (+1218222us) spinlock_profiling.cc:243] Waited 666 ms on lock 0x7b54000cb554. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:40.960149 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1532775808}
I20250905 19:59:41.042409 29645 consensus_queue.cc:1046] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20250905 19:59:41.009835 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:41.008416 28752 rpcz_store.cc:276] 0905 19:59:39.705367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:39.705548 (+ 181us) service_pool.cc:225] Handling call
0905 19:59:40.925467 (+1219919us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b54006fdbd4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:40.925585 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":379036928}
I20250905 19:59:40.991140 29810 raft_consensus.cc:513] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:41.011261 27908 catalog_manager.cc:5582] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 3 to 4, leader changed from cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 4 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:41.026145 29867 raft_consensus.cc:513] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.971328 28158 raft_consensus.cc:2466] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:59:40.993989 29804 leader_election.cc:290] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:40.996500 28076 raft_consensus.cc:2466] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:40.991303 29862 raft_consensus.cc:513] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.995554 28077 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
W20250905 19:59:40.962360 29713 consensus_meta.cc:287] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.559s user 0.005s sys 0.004s
I20250905 19:59:40.970760 29871 raft_consensus.cc:513] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:40.996165 29644 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:40.972995 29876 raft_consensus.cc:3058] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:41.116008 28079 raft_consensus.cc:2466] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:40.992336 29809 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:40.996676 29505 raft_consensus.cc:491] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
W20250905 19:59:41.063756 28081 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.710s user 0.029s sys 0.094s
W20250905 19:59:41.125341 28081 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.772s user 0.029s sys 0.095s
W20250905 19:59:41.066450 27973 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.580s user 0.023s sys 0.128s
I20250905 19:59:41.066260 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:41.126470 27973 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.640s user 0.023s sys 0.128s
W20250905 19:59:41.066885 28049 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.656s user 0.032s sys 0.038s
I20250905 19:59:41.126708 28080 raft_consensus.cc:2466] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
W20250905 19:59:41.068941 27975 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.614s user 0.015s sys 0.064s
I20250905 19:59:41.027325 29788 leader_election.cc:290] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:41.003504 28050 leader_election.cc:304] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:41.069156 27905 rpcz_store.cc:276] 0905 19:59:38.628496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:38.628674 (+ 178us) service_pool.cc:225] Handling call
0905 19:59:40.995340 (+2366666us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:59:39.731650 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:59:39.779268 (+ 47618us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:59:39.779323 (+ 55us) write_op.cc:435] Acquired schema lock
0905 19:59:39.779353 (+ 30us) tablet.cc:660] Decoding operations
0905 19:59:39.779905 (+ 552us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:59:39.780018 (+ 113us) write_op.cc:641] Partition lock acquired for write op
0905 19:59:39.780062 (+ 44us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:59:39.780489 (+ 427us) tablet.cc:699] Row locks acquired
0905 19:59:39.780515 (+ 26us) write_op.cc:260] PREPARE: finished
0905 19:59:39.780723 (+ 208us) write_op.cc:270] Start()
0905 19:59:39.780883 (+ 160us) write_op.cc:276] Timestamp: P: 1757102379780652 usec, L: 0
0905 19:59:39.780913 (+ 30us) op_driver.cc:348] REPLICATION: starting
0905 19:59:39.781820 (+ 907us) log.cc:844] Serialized 919 byte log entry
0905 19:59:40.332819 (+550999us) op_driver.cc:464] REPLICATION: finished
0905 19:59:40.744684 (+411865us) write_op.cc:301] APPLY: starting
0905 19:59:40.749756 (+ 5072us) tablet.cc:1365] starting BulkCheckPresence
0905 19:59:40.778186 (+ 28430us) tablet.cc:1368] finished BulkCheckPresence
0905 19:59:40.778322 (+ 136us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:59:40.856878 (+ 78556us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:59:40.910545 (+ 53667us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0905 19:59:40.917674 (+ 7129us) write_op.cc:312] APPLY: finished
0905 19:59:40.954922 (+ 37248us) log.cc:844] Serialized 29 byte log entry
0905 19:59:40.955103 (+ 181us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:59:40.955196 (+ 93us) write_op.cc:454] Released schema lock
0905 19:59:40.955238 (+ 42us) write_op.cc:341] FINISH: Updating metrics
0905 19:59:40.955656 (+ 418us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:59:40.955686 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":360953,"apply.run_cpu_time_us":2707,"apply.run_wall_time_us":229604,"num_ops":2,"prepare.queue_time_us":190648,"prepare.run_cpu_time_us":3954,"prepare.run_wall_time_us":89723,"replication_time_us":492847,"thread_start_us":180973,"threads_started":2}]]}
I20250905 19:59:41.076675 27904 catalog_manager.cc:5582] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 1 to 2, leader changed from e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 2 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:41.061677 29866 leader_election.cc:290] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:41.083027 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:41.087571 28123 leader_election.cc:304] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 19:59:41.085724 27908 catalog_manager.cc:5582] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 1 to 2, leader changed from cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 2 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:41.091931 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:41.036880 29800 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:41.127358 28049 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.720s user 0.033s sys 0.038s
I20250905 19:59:41.096675 29508 raft_consensus.cc:491] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:41.078868 29619 raft_consensus.cc:2802] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:41.115139 29804 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:41.134644 27908 catalog_manager.cc:5582] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 2 to 3, leader changed from cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 3 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
W20250905 19:59:41.135666 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 48220.9us
I20250905 19:59:41.134987 28078 raft_consensus.cc:2466] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:41.133278 28082 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 3.
I20250905 19:59:41.148702 28084 raft_consensus.cc:2466] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
W20250905 19:59:41.149384 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 799978us
I20250905 19:59:41.150259 29850 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:41.151078 29870 raft_consensus.cc:491] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:41.153514 29737 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:41.154060 29824 raft_consensus.cc:491] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:41.155753 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:41.156026 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:41.156113 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:41.156874 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:41.157284 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:41.159049 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:41.160400 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:41.161278 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:59:41.094844 28157 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.558s user 0.010s sys 0.034s
W20250905 19:59:41.235042 28157 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.700s user 0.010s sys 0.034s
I20250905 19:59:41.162876 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:41.163734 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:41.227669 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:41.236565 28009 raft_consensus.cc:2466] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:41.236981 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:41.229558 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:41.162182 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:41.238000 28005 raft_consensus.cc:2466] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 2.
I20250905 19:59:41.192068 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:41.193542 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 190134us
I20250905 19:59:41.195870 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:41.125418 29809 raft_consensus.cc:3058] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:41.213466 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:41.248291 28151 raft_consensus.cc:2466] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:41.233283 28153 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 19:59:41.202312 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:41.230765 27975 leader_election.cc:304] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:41.249061 28049 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:41.249465 28010 raft_consensus.cc:2391] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:41.203931 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:41.215821 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:41.250152 27975 leader_election.cc:304] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:41.250280 28078 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 19:59:41.127359 29505 raft_consensus.cc:3058] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:41.227813 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:41.256583 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 344775us
I20250905 19:59:41.256831 28082 raft_consensus.cc:2466] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 19:59:41.228140 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:41.233163 28007 raft_consensus.cc:2466] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 3.
I20250905 19:59:41.238705 28080 raft_consensus.cc:2466] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:41.234653 28150 raft_consensus.cc:2466] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:41.258231 28079 raft_consensus.cc:3058] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:41.258701 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:59:41.087337 29805 consensus_meta.cc:287] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.897s user 0.018s sys 0.003s
I20250905 19:59:41.246822 28158 raft_consensus.cc:2466] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:59:41.207304 29869 raft_consensus.cc:2802] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:41.251168 28003 raft_consensus.cc:2466] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
W20250905 19:59:41.206187 29749 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.730s user 0.024s sys 0.035s
I20250905 19:59:41.227526 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:41.251286 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:41.280298 28078 raft_consensus.cc:2466] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:59:41.254652 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:41.254928 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:41.281973 28010 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:41.122165 29713 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:41.256254 28154 raft_consensus.cc:2408] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20250905 19:59:41.247455 28050 leader_election.cc:304] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 19:59:41.251621 27976 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.701s user 0.014s sys 0.050s
I20250905 19:59:41.199959 29619 raft_consensus.cc:491] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:41.267776 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:41.207698 29818 raft_consensus.cc:491] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:41.211156 29738 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:41.270249 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:41.146258 29856 consensus_queue.cc:1035] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.106s
I20250905 19:59:41.260465 29875 raft_consensus.cc:491] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:41.201085 29757 raft_consensus.cc:2802] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:41.282136 28159 raft_consensus.cc:2466] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:41.318562 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:41.274823 29869 raft_consensus.cc:491] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250905 19:59:41.289033 27976 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.749s user 0.014s sys 0.050s
I20250905 19:59:41.292034 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:41.294900 27974 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:41.298339 28009 raft_consensus.cc:3058] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 3
I20250905 19:59:41.323215 28156 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 3.
I20250905 19:59:41.298022 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:41.299753 28050 leader_election.cc:304] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:41.301756 28077 raft_consensus.cc:3058] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 4
I20250905 19:59:41.327508 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:41.289480 29801 raft_consensus.cc:491] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:41.259781 28008 raft_consensus.cc:3058] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:41.199882 29508 raft_consensus.cc:513] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:41.302376 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:41.278581 28122 leader_election.cc:304] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:41.259960 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:41.276422 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:41.302299 29619 raft_consensus.cc:3058] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
W20250905 19:59:41.279871 29749 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.804s user 0.024s sys 0.035s
I20250905 19:59:41.211727 29804 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:41.414341 28009 raft_consensus.cc:2466] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
W20250905 19:59:41.231283 29864 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s user 0.005s sys 0.047s
I20250905 19:59:41.275329 29855 raft_consensus.cc:491] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:41.269371 29826 raft_consensus.cc:491] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:41.288482 29833 raft_consensus.cc:2802] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:41.331408 28050 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 19:59:41.421518 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 90208.2us
I20250905 19:59:41.253180 29737 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:41.300076 28717 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 19:59:41.302505 29852 raft_consensus.cc:2802] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:41.255077 29870 raft_consensus.cc:513] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:41.274930 29805 raft_consensus.cc:513] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:41.313412 29714 raft_consensus.cc:491] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:41.391940 28010 raft_consensus.cc:2466] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
W20250905 19:59:41.419816 29864 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.710s user 0.005s sys 0.047s
I20250905 19:59:41.233867 29850 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:41.394265 28080 raft_consensus.cc:2408] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250905 19:59:41.395359 28005 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:41.500860 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:41.501521 28084 raft_consensus.cc:2466] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 2.
I20250905 19:59:41.319212 29599 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 19:59:41.400090 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 121727us
I20250905 19:59:41.503818 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:41.328073 29869 raft_consensus.cc:3058] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:41.393654 29644 raft_consensus.cc:2802] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:41.327569 29756 leader_election.cc:290] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:41.332365 29854 leader_election.cc:290] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:41.416218 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:41.418762 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:41.320149 29818 raft_consensus.cc:513] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:41.323196 29738 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:41.399084 29800 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:41.447662 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 147694us
I20250905 19:59:41.424144 29867 leader_election.cc:290] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:41.426673 28050 leader_election.cc:304] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:41.398633 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:41.407917 29810 leader_election.cc:290] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:41.468068 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:41.255896 29824 raft_consensus.cc:513] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:41.325039 29875 raft_consensus.cc:513] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:41.487288 28125 leader_election.cc:304] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:41.407119 29853 leader_election.cc:290] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 19:59:41.474972 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 218433us
I20250905 19:59:41.575438 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:41.576143 28010 raft_consensus.cc:2466] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 19:59:41.577033 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 19:59:41.577277 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 150823us
I20250905 19:59:41.499822 28718 leader_election.cc:304] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:41.488058 28085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1290) took 1077 ms. Trace:
I20250905 19:59:41.504484 28749 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
W20250905 19:59:41.578263 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 91105.8us
I20250905 19:59:41.404450 29788 consensus_queue.cc:1035] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.065s
W20250905 19:59:41.473981 29787 consensus_meta.cc:287] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.631s user 0.000s sys 0.010s
I20250905 19:59:41.515998 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:41.622932 28005 raft_consensus.cc:2466] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:41.624332 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:41.625188 28005 raft_consensus.cc:2466] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:41.402009 29801 raft_consensus.cc:513] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:41.476106 29890 log.cc:927] Time spent T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.079s user 0.001s sys 0.000s
I20250905 19:59:41.446923 29862 leader_election.cc:290] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:41.520370 28006 raft_consensus.cc:2391] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
W20250905 19:59:41.477908 29877 log.cc:927] Time spent T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.188s user 0.000s sys 0.001s
I20250905 19:59:41.470211 29882 raft_consensus.cc:2747] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:41.424343 29833 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:41.470460 29822 raft_consensus.cc:2802] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 19:59:41.471582 29880 raft_consensus.cc:2802] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:41.472365 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:41.560761 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:41.561123 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:41.457512 29852 raft_consensus.cc:491] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:41.471819 29881 raft_consensus.cc:2802] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:41.467001 29813 raft_consensus.cc:2802] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250905 19:59:41.566372 29745 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.732s user 0.076s sys 0.140s
I20250905 19:59:41.578549 28085 rpcz_store.cc:276] 0905 19:59:40.410941 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:40.411229 (+ 288us) service_pool.cc:225] Handling call
0905 19:59:40.411635 (+ 406us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:40.810561 (+398926us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:40.810603 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:40.836023 (+ 25420us) log.cc:844] Serialized 26 byte log entry
0905 19:59:41.261427 (+425404us) raft_consensus.cc:1643] Marking committed up to 2
0905 19:59:41.261468 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:41.261569 (+ 101us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:41.487720 (+226151us) raft_consensus.cc:1683] finished
0905 19:59:41.487752 (+ 32us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:41.487977 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":166019,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":108,"thread_start_us":789640,"threads_started":2,"wal-append.queue_time_us":624957}
I20250905 19:59:41.609668 28050 leader_election.cc:304] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:41.610822 28003 raft_consensus.cc:2466] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 19:59:41.622182 29787 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:41.632933 28083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 158) took 1293 ms. Trace:
I20250905 19:59:41.641770 28083 rpcz_store.cc:276] 0905 19:59:40.340013 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:40.351321 (+ 11308us) service_pool.cc:225] Handling call
0905 19:59:40.368045 (+ 16724us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:40.405787 (+ 37742us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:40.405836 (+ 49us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:40.406495 (+ 659us) log.cc:844] Serialized 26 byte log entry
0905 19:59:40.834499 (+428004us) raft_consensus.cc:1643] Marking committed up to 6
0905 19:59:40.853951 (+ 19452us) log.cc:844] Serialized 14 byte log entry
0905 19:59:40.854272 (+ 321us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:40.854401 (+ 129us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:41.632598 (+778197us) raft_consensus.cc:1683] finished
0905 19:59:41.632631 (+ 33us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:41.632848 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":65959,"prepare.queue_time_us":3553,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":38,"raft.queue_time_us":182486,"raft.run_cpu_time_us":333,"raft.run_wall_time_us":23125,"thread_start_us":412853,"threads_started":1,"wal-append.queue_time_us":804506}
I20250905 19:59:41.633210 29833 raft_consensus.cc:3058] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:41.633406 29822 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:41.647209 29822 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 19:59:41.633582 29880 raft_consensus.cc:491] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:41.651243 29880 raft_consensus.cc:3058] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:41.634176 28744 raft_consensus.cc:2366] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4.
I20250905 19:59:41.634613 28009 raft_consensus.cc:2466] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:41.657779 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:41.634887 29852 raft_consensus.cc:3058] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:41.634924 28751 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
I20250905 19:59:41.635058 29881 raft_consensus.cc:491] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:41.664440 29881 raft_consensus.cc:3058] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:41.635202 29813 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:41.667225 29813 raft_consensus.cc:3058] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Advancing to term 4
W20250905 19:59:41.635453 29745 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.802s user 0.077s sys 0.140s
I20250905 19:59:41.635435 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:41.673808 28006 raft_consensus.cc:2466] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 19:59:41.675169 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:41.675936 28006 raft_consensus.cc:2466] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 19:59:41.628595 29801 leader_election.cc:290] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:41.686249 28009 raft_consensus.cc:3058] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:41.626420 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:41.703608 28005 raft_consensus.cc:2466] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:41.640933 29787 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:41.744810 29822 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:41.468243 29826 raft_consensus.cc:513] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:41.423843 29855 raft_consensus.cc:513] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:41.620064 28125 leader_election.cc:304] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:41.593487 28001 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 1.
W20250905 19:59:41.768285 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 158714us
I20250905 19:59:41.954133 28050 leader_election.cc:304] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 19:59:41.748934 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 249198us
W20250905 19:59:41.749027 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 183629us
I20250905 19:59:41.955678 28717 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 19:59:41.573156 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 322138us
I20250905 19:59:41.502969 29714 raft_consensus.cc:513] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:41.566672 29644 raft_consensus.cc:695] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:41.509689 29873 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:41.567919 29840 raft_consensus.cc:491] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:41.575528 29871 leader_election.cc:290] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:41.606822 28746 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:59:41.800177 29805 leader_election.cc:290] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:41.617132 29878 raft_consensus.cc:2802] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:41.975303 29901 raft_consensus.cc:2802] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:41.607088 29511 raft_consensus.cc:491] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:41.748467 29867 raft_consensus.cc:491] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:41.864923 27974 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.015s sys 0.051s
W20250905 19:59:42.091527 27974 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.768s user 0.016s sys 0.051s
W20250905 19:59:41.963055 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 343045us
I20250905 19:59:42.093464 28125 leader_election.cc:304] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:41.870627 28049 leader_election.cc:400] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299) with higher term. Message: Invalid argument: T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4.
I20250905 19:59:42.094376 28049 leader_election.cc:403] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
W20250905 19:59:41.871441 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 367094us
I20250905 19:59:42.096796 28122 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:41.923581 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:41.762341 28009 raft_consensus.cc:2466] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:41.968008 27975 leader_election.cc:304] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
W20250905 19:59:41.975680 28007 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.691s user 0.019s sys 0.082s
W20250905 19:59:42.102100 28007 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.817s user 0.019s sys 0.082s
W20250905 19:59:41.803377 28079 consensus_meta.cc:287] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.528s user 0.010s sys 0.000s
W20250905 19:59:42.102519 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 134637us
I20250905 19:59:42.103150 28079 raft_consensus.cc:2466] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 19:59:42.103953 27975 leader_election.cc:304] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:42.105425 28715 leader_election.cc:304] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:42.105721 27975 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:42.106901 27975 leader_election.cc:304] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:42.107911 27975 leader_election.cc:304] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:42.110376 28122 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:41.980633 28718 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:42.082657 29878 raft_consensus.cc:491] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:41.617379 29864 consensus_queue.cc:1035] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.193s
W20250905 19:59:41.783068 29885 log.cc:927] Time spent T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.278s user 0.001s sys 0.000s
I20250905 19:59:41.969023 29801 raft_consensus.cc:491] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
W20250905 19:59:41.784878 29887 log.cc:927] Time spent T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.216s user 0.000s sys 0.000s
W20250905 19:59:41.783629 29891 log.cc:927] Time spent T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.314s user 0.000s sys 0.000s
W20250905 19:59:41.800457 29505 consensus_meta.cc:287] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.521s user 0.007s sys 0.003s
I20250905 19:59:41.767138 29888 raft_consensus.cc:2802] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:42.123111 28155 raft_consensus.cc:2466] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:59:42.124213 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:41.775751 29810 raft_consensus.cc:491] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:41.775602 29849 raft_consensus.cc:491] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
W20250905 19:59:42.091939 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 797075us
I20250905 19:59:42.128314 28125 leader_election.cc:304] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:42.135367 29878 raft_consensus.cc:3058] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:41.776346 29853 raft_consensus.cc:2802] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 19:59:41.770062 29800 raft_consensus.cc:2802] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:41.767413 29854 raft_consensus.cc:2802] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:41.793130 29848 raft_consensus.cc:3053] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Stepping down as leader of term 3
W20250905 19:59:41.870654 29841 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.560s user 0.040s sys 0.070s
I20250905 19:59:41.777644 29713 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:42.083489 29901 raft_consensus.cc:491] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:41.812827 29738 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:41.795634 29895 raft_consensus.cc:491] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:41.799070 29804 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:41.951558 29508 leader_election.cc:290] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:41.795471 29599 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:42.084086 29757 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.694s user 0.041s sys 0.085s
W20250905 19:59:42.143097 27976 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.812s user 0.047s sys 0.083s
W20250905 19:59:42.219887 27976 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.897s user 0.048s sys 0.084s
I20250905 19:59:41.826094 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:42.220705 28750 raft_consensus.cc:3058] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:41.801836 29818 leader_election.cc:290] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:41.794521 29850 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:41.805644 28745 raft_consensus.cc:1273] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250905 19:59:41.973562 28076 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1327) took 1111 ms. Trace:
I20250905 19:59:41.795353 29749 raft_consensus.cc:491] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
W20250905 19:59:41.981480 28082 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.008s sys 0.015s
W20250905 19:59:42.226545 28082 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.756s user 0.008s sys 0.016s
I20250905 19:59:42.131294 29805 raft_consensus.cc:2802] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250905 19:59:42.161742 28008 consensus_meta.cc:287] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.762s user 0.010s sys 0.000s
I20250905 19:59:42.171602 29813 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:42.229318 28008 raft_consensus.cc:2466] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 19:59:42.184331 29505 raft_consensus.cc:513] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:42.166285 29899 raft_consensus.cc:2747] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:42.134443 29862 raft_consensus.cc:2802] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:42.134142 29871 raft_consensus.cc:2802] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election won for term 5
I20250905 19:59:42.174623 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:42.189574 29810 raft_consensus.cc:513] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:41.971150 29883 consensus_queue.cc:1046] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.184s
I20250905 19:59:41.961222 29852 raft_consensus.cc:513] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:42.250815 29873 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:42.252681 29852 leader_election.cc:290] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:42.140465 29788 raft_consensus.cc:2802] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250905 19:59:42.185801 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 57610.4us
I20250905 19:59:42.253582 29788 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:42.253964 29788 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Advancing to term 7
I20250905 19:59:42.254523 28125 leader_election.cc:304] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:42.255455 28125 leader_election.cc:304] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 19:59:42.188155 27973 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.020s sys 0.023s
I20250905 19:59:42.091024 29840 raft_consensus.cc:513] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:42.201807 29880 consensus_meta.cc:287] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.550s user 0.012s sys 0.001s
I20250905 19:59:42.186944 29870 leader_election.cc:290] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:42.192438 29833 consensus_meta.cc:287] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.547s user 0.012s sys 0.000s
I20250905 19:59:42.132179 29867 raft_consensus.cc:513] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:42.131946 29511 raft_consensus.cc:513] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:42.183511 29737 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:42.190485 29756 raft_consensus.cc:2802] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 19:59:42.189951 29800 raft_consensus.cc:491] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:42.188596 29853 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:42.191697 29787 raft_consensus.cc:2747] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:42.189728 29854 raft_consensus.cc:491] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:42.196698 29848 raft_consensus.cc:738] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:42.159778 29801 raft_consensus.cc:513] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:42.229898 28008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 174) took 1071 ms. Trace:
I20250905 19:59:42.238049 29749 raft_consensus.cc:513] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:42.236366 28157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 1880) took 2008 ms. Trace:
I20250905 19:59:42.287418 29873 raft_consensus.cc:2802] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:42.189900 29849 raft_consensus.cc:513] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:42.222322 29757 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.833s user 0.041s sys 0.085s
W20250905 19:59:42.225618 29881 consensus_meta.cc:287] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.561s user 0.009s sys 0.005s
I20250905 19:59:42.243804 28752 raft_consensus.cc:2466] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 19:59:42.323338 28752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40926 (request call id 376) took 1003 ms. Trace:
I20250905 19:59:42.323810 28752 rpcz_store.cc:276] 0905 19:59:41.320276 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:41.320885 (+ 609us) service_pool.cc:225] Handling call
0905 19:59:41.522925 (+202040us) spinlock_profiling.cc:243] Waited 946 us on lock 0x7b5800304598. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b7db60 00007fe4a2b7d444 00007fe49c2050c4 00007fe49c205052
0905 19:59:41.804517 (+281592us) spinlock_profiling.cc:243] Waited 6.05 ms on lock 0x7b5800304598. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac6721e 000000000035a3c9 00007fe4a2b08310 00007fe4a2bceed4 00007fe4a2bcef41 00007fe4a2b7db60 00007fe4a2b7d444 00007fe49c2050c4 00007fe49c205052
0905 19:59:42.174487 (+369970us) spinlock_profiling.cc:243] Waited 32.6 ms on lock 0x7b5800304598. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac672bc 00007fe49ac67226 000000000035a3c9 00007fe4a2b08310 00007fe4a2bceed4 00007fe4a2bcef41 00007fe4a2b7db60 00007fe4a2b7d444 00007fe49c2050c4 00007fe49c205052
0905 19:59:42.323252 (+148765us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":91083776}
I20250905 19:59:42.129369 28122 leader_election.cc:304] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 19:59:42.325670 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 196385us
I20250905 19:59:42.220570 29824 leader_election.cc:290] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:42.219635 29713 raft_consensus.cc:2802] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:42.219192 29804 raft_consensus.cc:2802] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:42.222011 29855 leader_election.cc:290] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:42.248764 29905 raft_consensus.cc:2802] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250905 19:59:42.249734 29902 log.cc:927] Time spent T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.108s user 0.000s sys 0.001s
I20250905 19:59:42.220101 29900 raft_consensus.cc:2802] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250905 19:59:42.256383 27973 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.587s user 0.020s sys 0.023s
I20250905 19:59:42.219389 29508 raft_consensus.cc:491] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
W20250905 19:59:42.346804 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 677739us
I20250905 19:59:42.319686 29757 raft_consensus.cc:695] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:42.261689 28125 leader_election.cc:304] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 19:59:42.351485 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 89876.4us
I20250905 19:59:42.228996 29805 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:42.240833 29850 raft_consensus.cc:491] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:42.284054 28005 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.032s sys 0.022s
W20250905 19:59:42.247742 29815 consensus_meta.cc:287] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 1.156s user 0.010s sys 0.003s
W20250905 19:59:42.288893 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 330498us
I20250905 19:59:42.241518 29903 raft_consensus.cc:491] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:42.346597 29900 raft_consensus.cc:491] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:42.202535 29644 consensus_queue.cc:237] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:42.225484 29599 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:42.250847 29883 raft_consensus.cc:3053] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 LEADER]: Stepping down as leader of term 4
I20250905 19:59:42.218308 29895 raft_consensus.cc:513] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:42.261528 29862 raft_consensus.cc:695] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:42.166659 29894 raft_consensus.cc:2802] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:42.260681 29875 leader_election.cc:290] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:42.308885 28008 rpcz_store.cc:276] 0905 19:59:41.158567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:41.158729 (+ 162us) service_pool.cc:225] Handling call
0905 19:59:42.229834 (+1071105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:42.207443 29901 raft_consensus.cc:3058] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:42.417207 28750 raft_consensus.cc:2466] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:42.417603 29810 leader_election.cc:290] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:42.363447 29815 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:42.357432 29508 raft_consensus.cc:513] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:42.310137 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 354537us
I20250905 19:59:42.419910 28717 leader_election.cc:304] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:42.420508 29815 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:42.421263 29815 raft_consensus.cc:2802] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 19:59:42.421664 29815 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:42.422016 29815 raft_consensus.cc:3058] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Advancing to term 6
W20250905 19:59:42.318715 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 210882us
W20250905 19:59:42.307296 29876 consensus_meta.cc:287] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 1.180s user 0.018s sys 0.000s
I20250905 19:59:42.283236 29904 raft_consensus.cc:2802] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:42.312857 28157 rpcz_store.cc:276] 0905 19:59:40.227992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:40.294032 (+ 66040us) service_pool.cc:225] Handling call
0905 19:59:40.346362 (+ 52330us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:40.436646 (+ 90284us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:40.436696 (+ 50us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:40.524017 (+ 87321us) log.cc:844] Serialized 26 byte log entry
0905 19:59:41.235489 (+711472us) raft_consensus.cc:1643] Marking committed up to 2
0905 19:59:41.235544 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:41.235639 (+ 95us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:42.236069 (+1000430us) raft_consensus.cc:1683] finished
0905 19:59:42.236103 (+ 34us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:42.236286 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8227,"raft.queue_time_us":75325,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":10518,"thread_start_us":774810,"threads_started":2,"wal-append.queue_time_us":885096}
I20250905 19:59:42.292423 29756 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:42.329075 28122 leader_election.cc:304] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:42.307996 29800 raft_consensus.cc:3058] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:42.304970 29822 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:42.333947 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 228607us
I20250905 19:59:42.305456 29826 leader_election.cc:290] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:42.288506 29880 raft_consensus.cc:513] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:42.306028 29854 raft_consensus.cc:3058] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:42.307571 29853 raft_consensus.cc:3058] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:42.222595 28076 rpcz_store.cc:276] 0905 19:59:40.855243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:41.508736 (+653493us) service_pool.cc:225] Handling call
0905 19:59:41.773781 (+265045us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:41.840627 (+ 66846us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:41.843911 (+ 3284us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:41.866248 (+ 22337us) raft_consensus.cc:1643] Marking committed up to 2
0905 19:59:41.870133 (+ 3885us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:41.919459 (+ 49326us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:41.966741 (+ 47282us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:42.308945 29714 leader_election.cc:290] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 19:59:42.321635 29809 consensus_meta.cc:287] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.054s user 0.013s sys 0.000s
I20250905 19:59:42.290479 29833 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:42.318380 29873 raft_consensus.cc:491] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:42.219594 29738 raft_consensus.cc:491] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:42.337992 29787 raft_consensus.cc:2802] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250905 19:59:42.353668 28003 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.017s sys 0.013s
I20250905 19:59:42.338536 29713 raft_consensus.cc:491] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:42.359756 28123 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
W20250905 19:59:42.361332 28005 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.624s user 0.032s sys 0.023s
I20250905 19:59:42.369722 29850 raft_consensus.cc:513] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:42.357064 29805 raft_consensus.cc:3058] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
W20250905 19:59:42.352031 29898 log.cc:927] Time spent T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
W20250905 19:59:42.290129 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 336090us
I20250905 19:59:42.348367 29905 raft_consensus.cc:491] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:42.347299 29804 raft_consensus.cc:695] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:42.339107 29881 raft_consensus.cc:513] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:42.391769 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 65097.9us
W20250905 19:59:42.392752 29911 log.cc:927] Time spent T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20250905 19:59:42.399945 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:42.403549 29894 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:42.507822 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:42.403757 29906 raft_consensus.cc:491] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:42.405139 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
W20250905 19:59:42.406582 29892 log.cc:927] Time spent T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.469s user 0.000s sys 0.002s
I20250905 19:59:42.407729 29849 leader_election.cc:290] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:42.409158 29813 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:42.348995 29848 consensus_queue.cc:260] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:42.281869 29871 raft_consensus.cc:695] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:42.426872 28151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1417) took 1209 ms. Trace:
I20250905 19:59:42.427572 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 19:59:42.461406 29869 consensus_meta.cc:287] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.935s user 0.011s sys 0.000s
I20250905 19:59:42.428997 28718 leader_election.cc:304] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:42.361639 29645 raft_consensus.cc:2802] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:42.442369 27975 leader_election.cc:304] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 19:59:42.615831 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 173577us
I20250905 19:59:42.373426 29883 raft_consensus.cc:738] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
W20250905 19:59:42.503180 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 636103us
I20250905 19:59:42.490917 28078 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1323) took 1640 ms. Trace:
I20250905 19:59:42.508921 29713 raft_consensus.cc:3058] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:42.507360 29905 raft_consensus.cc:3058] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:42.510675 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:42.512158 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:42.632781 28079 raft_consensus.cc:2391] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 4: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:42.480428 28749 raft_consensus.cc:1273] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250905 19:59:42.512928 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:42.385501 29866 raft_consensus.cc:491] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:42.508414 27973 leader_election.cc:304] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:42.513674 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:42.374778 29900 raft_consensus.cc:3058] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:42.597810 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 19:59:42.647987 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 135629us
I20250905 19:59:42.648679 28157 raft_consensus.cc:3058] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
W20250905 19:59:42.599953 28001 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 171) took 1444 ms (client timeout 1607 ms). Trace:
I20250905 19:59:42.649119 28125 leader_election.cc:304] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 19:59:42.488200 28077 consensus_meta.cc:287] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 1.092s user 0.008s sys 0.004s
I20250905 19:59:42.460539 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:42.374117 29903 raft_consensus.cc:513] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:42.602653 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:42.607090 28008 raft_consensus.cc:2391] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:42.428833 28751 raft_consensus.cc:2391] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 4: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
W20250905 19:59:42.463653 29619 consensus_meta.cc:287] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.042s user 0.006s sys 0.006s
I20250905 19:59:42.609112 28006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1221) took 1356 ms. Trace:
I20250905 19:59:42.463591 29901 raft_consensus.cc:513] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:42.609712 28010 raft_consensus.cc:2366] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3.
I20250905 19:59:42.610181 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:42.420535 29508 leader_election.cc:290] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:42.485965 29867 leader_election.cc:290] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:42.659770 29745 consensus_queue.cc:1035] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20250905 19:59:42.602465 27892 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 986c4f475a8d46f39465775db70416d3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20250905 19:59:42.715499 27892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 209856us
I20250905 19:59:42.508805 29787 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:42.624998 28049 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:42.716727 29787 raft_consensus.cc:3058] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
W20250905 19:59:42.716990 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 92078.4us
I20250905 19:59:42.718247 28049 leader_election.cc:304] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:42.719481 29622 raft_consensus.cc:2802] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 19:59:42.719903 29622 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:42.509150 29873 raft_consensus.cc:3058] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:42.720273 29622 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:42.669732 28077 raft_consensus.cc:2466] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
W20250905 19:59:42.721444 28077 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50660 (request call id 162) took 1623 ms (client timeout 1844 ms). Trace:
W20250905 19:59:42.721859 28077 rpcz_store.cc:269] 0905 19:59:41.104411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:41.144163 (+ 39752us) service_pool.cc:225] Handling call
0905 19:59:42.721377 (+1577214us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:42.723364 29833 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:42.723852 29888 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:42.724315 29888 raft_consensus.cc:3058] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:42.725021 29787 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:42.461710 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:42.726666 28745 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Advancing to term 6
I20250905 19:59:42.726500 29619 raft_consensus.cc:513] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:42.727265 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:42.728917 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:42.504904 28003 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.671s user 0.018s sys 0.013s
I20250905 19:59:42.595748 29756 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 19:59:42.464025 29505 leader_election.cc:290] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:42.509763 28752 raft_consensus.cc:2366] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 3. Current term is 4.
I20250905 19:59:42.441315 29904 raft_consensus.cc:491] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:42.480118 29810 raft_consensus.cc:491] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:42.515273 29894 raft_consensus.cc:3058] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:42.595075 29840 leader_election.cc:290] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:42.624959 28078 rpcz_store.cc:276] 0905 19:59:40.850649 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:41.280983 (+430334us) service_pool.cc:225] Handling call
0905 19:59:41.281364 (+ 381us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:41.305804 (+ 24440us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:41.305837 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:41.306297 (+ 460us) log.cc:844] Serialized 26 byte log entry
0905 19:59:41.603275 (+296978us) raft_consensus.cc:1643] Marking committed up to 1
0905 19:59:41.603342 (+ 67us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:41.603447 (+ 105us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:42.490527 (+887080us) raft_consensus.cc:1683] finished
0905 19:59:42.490565 (+ 38us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:42.490740 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15961,"raft.queue_time_us":20962,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":13128,"thread_start_us":276040,"threads_started":1,"wal-append.queue_time_us":628394}
W20250905 19:59:42.643220 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 134867us
W20250905 19:59:42.649408 28001 rpcz_store.cc:269] 0905 19:59:41.155665 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:41.155785 (+ 120us) service_pool.cc:225] Handling call
0905 19:59:42.599686 (+1443901us) spinlock_profiling.cc:243] Waited 295 ms on lock 0x7b5400383bd4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:42.599867 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":677480832}
I20250905 19:59:42.749161 28081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 155) took 2482 ms. Trace:
I20250905 19:59:42.609017 29714 raft_consensus.cc:2747] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 19:59:42.662256 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:42.761384 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:42.441120 29824 consensus_queue.cc:1035] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.187s
I20250905 19:59:42.615399 29645 raft_consensus.cc:491] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:42.609220 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:42.628027 29818 raft_consensus.cc:2802] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:42.681005 28006 rpcz_store.cc:276] 0905 19:59:41.252388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:41.676668 (+424280us) service_pool.cc:225] Handling call
0905 19:59:41.677091 (+ 423us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:41.677531 (+ 440us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:41.677574 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:41.678202 (+ 628us) log.cc:844] Serialized 26 byte log entry
0905 19:59:42.171621 (+493419us) raft_consensus.cc:1643] Marking committed up to 1
0905 19:59:42.171684 (+ 63us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:42.171783 (+ 99us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:42.608851 (+437068us) raft_consensus.cc:1683] finished
0905 19:59:42.608874 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:42.609055 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":275209,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":19831,"thread_start_us":449264,"threads_started":1,"wal-append.queue_time_us":610590}
I20250905 19:59:42.707998 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:42.771178 28084 raft_consensus.cc:2391] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 6: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:42.708984 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:42.772368 28079 raft_consensus.cc:2466] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:42.714422 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:42.605674 28151 rpcz_store.cc:276] 0905 19:59:41.217437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:41.259700 (+ 42263us) service_pool.cc:225] Handling call
0905 19:59:41.288469 (+ 28769us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:41.611545 (+323076us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:41.611585 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:41.775700 (+164115us) log.cc:844] Serialized 26 byte log entry
0905 19:59:42.222056 (+446356us) raft_consensus.cc:1643] Marking committed up to 2
0905 19:59:42.385066 (+163010us) log.cc:844] Serialized 14 byte log entry
0905 19:59:42.426476 (+ 41410us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:42.426561 (+ 85us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:42.426607 (+ 46us) raft_consensus.cc:1683] finished
0905 19:59:42.426621 (+ 14us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:42.426735 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18372,"prepare.queue_time_us":120890,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":1997,"raft.queue_time_us":138320,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":18135,"thread_start_us":905117,"threads_started":4,"wal-append.queue_time_us":538020}
I20250905 19:59:42.727192 29787 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:42.514897 29738 raft_consensus.cc:513] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:42.731968 28155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33752 (request call id 1409) took 1576 ms. Trace:
I20250905 19:59:42.781867 28155 rpcz_store.cc:276] 0905 19:59:41.156027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:41.156617 (+ 590us) service_pool.cc:225] Handling call
0905 19:59:42.731825 (+1575208us) spinlock_profiling.cc:243] Waited 889 ms on lock 0x7b5400200b54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:42.731916 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2044833024}
I20250905 19:59:42.460403 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:42.729698 28083 raft_consensus.cc:2466] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:42.514601 29809 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:42.641577 29801 leader_election.cc:290] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:42.653070 29749 leader_election.cc:290] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:42.652766 29848 raft_consensus.cc:3058] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:42.745342 28009 raft_consensus.cc:2466] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 1.
I20250905 19:59:42.766209 29505 raft_consensus.cc:2802] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 19:59:42.748718 29822 raft_consensus.cc:2802] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:42.754663 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:42.755772 29850 leader_election.cc:290] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:42.625914 29869 raft_consensus.cc:513] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:42.757932 27973 leader_election.cc:304] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
W20250905 19:59:42.819543 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 61703.4us
I20250905 19:59:42.820706 27973 leader_election.cc:304] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:42.762815 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:42.765826 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:42.670390 29862 consensus_queue.cc:237] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:42.758273 28081 rpcz_store.cc:276] 0905 19:59:40.258029 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:40.258300 (+ 271us) service_pool.cc:225] Handling call
0905 19:59:40.295219 (+ 36919us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:41.125905 (+830686us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:41.125944 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:41.145448 (+ 19504us) log.cc:844] Serialized 26 byte log entry
0905 19:59:41.573955 (+428507us) raft_consensus.cc:1643] Marking committed up to 4
0905 19:59:41.604491 (+ 30536us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:41.604593 (+ 102us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:42.739571 (+1134978us) raft_consensus.cc:1683] finished
0905 19:59:42.739591 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:42.739785 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14470,"raft.queue_time_us":138865,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":119,"thread_start_us":1116085,"threads_started":2,"wal-append.queue_time_us":672750}
I20250905 19:59:42.756117 29867 raft_consensus.cc:491] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
W20250905 19:59:42.769726 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 440739us
I20250905 19:59:42.834873 27973 leader_election.cc:304] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:42.665393 29866 raft_consensus.cc:513] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:42.783631 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:42.743543 29904 raft_consensus.cc:3058] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:42.792114 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:42.853091 28083 raft_consensus.cc:2466] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:42.481951 29757 consensus_queue.cc:237] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:42.761467 29875 raft_consensus.cc:491] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:42.768898 29849 raft_consensus.cc:2747] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:42.870105 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:42.771245 29813 raft_consensus.cc:2802] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:42.805727 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:42.807294 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:42.765774 29508 raft_consensus.cc:491] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:42.776768 29818 raft_consensus.cc:491] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:42.777309 29645 raft_consensus.cc:3058] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:42.800766 28746 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 1.
I20250905 19:59:42.753633 29918 raft_consensus.cc:491] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:42.812776 28157 raft_consensus.cc:2466] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:42.762622 28152 raft_consensus.cc:2466] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:59:42.767578 28080 raft_consensus.cc:2408] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20250905 19:59:42.822233 28078 raft_consensus.cc:2466] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:42.627990 29906 raft_consensus.cc:513] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:42.824555 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:42.757922 29810 raft_consensus.cc:513] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:42.710283 29804 consensus_queue.cc:237] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:42.800825 29787 raft_consensus.cc:2802] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:42.846623 29809 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:42.832278 29822 raft_consensus.cc:695] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:42.849543 28743 raft_consensus.cc:1273] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250905 19:59:42.815183 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:42.801355 29919 raft_consensus.cc:491] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:42.860335 27973 leader_election.cc:304] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:42.860952 29931 raft_consensus.cc:491] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:42.861104 28002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 168) took 2607 ms. Trace:
I20250905 19:59:42.812649 29738 leader_election.cc:290] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:43.091454 29822 consensus_queue.cc:237] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:43.092320 29931 raft_consensus.cc:513] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:42.808359 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:42.809156 29505 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:42.962663 29930 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:42.867950 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:43.129184 28083 raft_consensus.cc:2466] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:42.845561 29867 raft_consensus.cc:513] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:42.819824 29901 leader_election.cc:290] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:42.447575 29876 raft_consensus.cc:513] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:42.871011 28081 raft_consensus.cc:2391] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 3: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
W20250905 19:59:42.869675 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 440720us
I20250905 19:59:42.833673 29850 raft_consensus.cc:2802] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250905 19:59:42.878489 28010 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1230) took 1394 ms (client timeout 1740 ms). Trace:
I20250905 19:59:42.852896 29745 raft_consensus.cc:2747] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:42.855172 29831 raft_consensus.cc:2747] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:42.778947 29871 consensus_queue.cc:237] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:43.165867 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 19:59:42.886957 29922 log.cc:927] Time spent T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.138s user 0.001s sys 0.000s
W20250905 19:59:42.873325 29921 log.cc:927] Time spent T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.425s user 0.002s sys 0.000s
I20250905 19:59:43.048810 28006 raft_consensus.cc:2466] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
I20250905 19:59:43.168794 28007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1216) took 1949 ms. Trace:
I20250905 19:59:43.168920 28003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1235) took 1683 ms. Trace:
I20250905 19:59:43.169492 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:43.170528 28006 raft_consensus.cc:2391] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 6: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:43.050881 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:43.097889 29886 raft_consensus.cc:2747] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299) with higher term. Message: Invalid argument: T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4.
I20250905 19:59:42.875664 29881 leader_election.cc:290] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:43.079203 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:59:43.084853 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 224598us
I20250905 19:59:42.888497 29818 raft_consensus.cc:3058] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:43.100275 28155 raft_consensus.cc:2408] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250905 19:59:42.955823 29813 raft_consensus.cc:695] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:43.085816 28002 rpcz_store.cc:276] 0905 19:59:40.245364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:40.245557 (+ 193us) service_pool.cc:225] Handling call
0905 19:59:40.285408 (+ 39851us) raft_consensus.cc:1469] Updating replica for 2 ops
0905 19:59:40.711607 (+426199us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:40.711642 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0905 19:59:40.725608 (+ 13966us) log.cc:844] Serialized 52 byte log entry
0905 19:59:41.280869 (+555261us) raft_consensus.cc:1643] Marking committed up to 4
0905 19:59:41.280932 (+ 63us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:41.281038 (+ 106us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:42.852663 (+1571625us) raft_consensus.cc:1683] finished
0905 19:59:42.852688 (+ 25us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:42.852840 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":57873,"raft.queue_time_us":77668,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":23823,"spinlock_wait_cycles":1792,"thread_start_us":832738,"threads_started":2,"wal-append.queue_time_us":825349}
I20250905 19:59:42.820554 29599 leader_election.cc:290] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:43.050391 29787 raft_consensus.cc:491] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:42.890137 29875 raft_consensus.cc:513] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:43.164075 29850 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:43.138926 28747 raft_consensus.cc:2391] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 4: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 19:59:43.059566 29895 leader_election.cc:290] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:43.165323 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:43.169967 28004 raft_consensus.cc:2391] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 4: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:43.171581 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:43.052140 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:43.267427 28080 raft_consensus.cc:2466] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:42.891880 29824 raft_consensus.cc:491] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:43.267829 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:43.268451 28743 raft_consensus.cc:3058] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:43.268371 29824 raft_consensus.cc:513] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:43.094054 29917 raft_consensus.cc:2802] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250905 19:59:43.163038 28010 rpcz_store.cc:269] 0905 19:59:41.483579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:41.576798 (+ 93219us) service_pool.cc:225] Handling call
0905 19:59:42.878150 (+1301352us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b5400290754. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:42.878375 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2373171584}
I20250905 19:59:43.269834 29917 raft_consensus.cc:491] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:43.270270 29917 raft_consensus.cc:3058] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:43.270486 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:43.169245 28003 rpcz_store.cc:276] 0905 19:59:41.485265 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:41.831523 (+346258us) service_pool.cc:225] Handling call
0905 19:59:41.832020 (+ 497us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:41.832473 (+ 453us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:41.832509 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:41.833215 (+ 706us) log.cc:844] Serialized 26 byte log entry
0905 19:59:42.730564 (+897349us) raft_consensus.cc:1643] Marking committed up to 1
0905 19:59:42.730627 (+ 63us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:42.730725 (+ 98us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:43.168718 (+437993us) raft_consensus.cc:1683] finished
0905 19:59:43.168746 (+ 28us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:43.168871 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":149234,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":55169,"thread_start_us":896838,"threads_started":1,"wal-append.queue_time_us":655226}
I20250905 19:59:43.104439 28748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40904 (request call id 484) took 1801 ms. Trace:
I20250905 19:59:43.091550 29903 leader_election.cc:290] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:43.199584 28157 raft_consensus.cc:2466] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
I20250905 19:59:43.031329 29873 raft_consensus.cc:513] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:43.254944 29787 raft_consensus.cc:3058] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
W20250905 19:59:43.139806 29893 log.cc:927] Time spent T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.837s user 0.005s sys 0.000s
W20250905 19:59:43.189126 28123 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.594s user 0.095s sys 0.134s
I20250905 19:59:43.169202 28007 rpcz_store.cc:276] 0905 19:59:41.219094 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:41.260317 (+ 41223us) service_pool.cc:225] Handling call
0905 19:59:41.260644 (+ 327us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:42.102582 (+841938us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:42.102621 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:42.131842 (+ 29221us) log.cc:844] Serialized 26 byte log entry
0905 19:59:42.305586 (+173744us) raft_consensus.cc:1643] Marking committed up to 1
0905 19:59:42.310380 (+ 4794us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:42.311596 (+ 1216us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:43.168513 (+856917us) raft_consensus.cc:1683] finished
0905 19:59:43.168542 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:43.168695 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16642,"raft.queue_time_us":123258,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":9629,"thread_start_us":937610,"threads_started":2,"wal-append.queue_time_us":191225}
I20250905 19:59:43.139483 29928 raft_consensus.cc:491] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:43.342948 29928 raft_consensus.cc:513] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:43.345208 29928 leader_election.cc:290] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:43.346900 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:43.347060 29810 leader_election.cc:290] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:43.347589 28157 raft_consensus.cc:2466] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
I20250905 19:59:43.348205 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:43.348510 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:43.349630 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:43.135424 29919 raft_consensus.cc:513] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:43.138684 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:43.351662 28750 raft_consensus.cc:2466] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:43.144212 29619 leader_election.cc:290] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 19:59:43.356957 29894 consensus_meta.cc:287] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.587s user 0.001s sys 0.012s
I20250905 19:59:43.052917 29849 raft_consensus.cc:491] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:43.127589 28077 raft_consensus.cc:2466] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:43.357753 29919 leader_election.cc:290] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:43.357415 29894 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:43.252401 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:42.966024 29508 raft_consensus.cc:513] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:43.275703 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:43.362119 28003 raft_consensus.cc:2466] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:43.282258 28081 raft_consensus.cc:2391] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 2: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:43.282410 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:59:43.207942 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:43.330924 29932 log.cc:927] Time spent T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.569s user 0.002s sys 0.000s
I20250905 19:59:43.266212 29876 leader_election.cc:290] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 19:59:43.340330 28123 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.745s user 0.095s sys 0.134s
I20250905 19:59:43.342180 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:43.127626 29505 raft_consensus.cc:3058] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
I20250905 19:59:43.356607 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:43.357583 29849 raft_consensus.cc:513] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:43.199231 29745 raft_consensus.cc:491] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:43.358743 28006 raft_consensus.cc:3058] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 3
I20250905 19:59:43.086628 29918 raft_consensus.cc:513] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:43.360447 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 302622us
I20250905 19:59:43.257726 29850 raft_consensus.cc:3058] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
W20250905 19:59:43.364374 28005 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.861s user 0.136s sys 0.182s
W20250905 19:59:43.329867 28716 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.840s user 0.006s sys 0.006s
I20250905 19:59:43.365348 28150 raft_consensus.cc:2391] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:43.358165 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:43.374644 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 1.01488e+06us
I20250905 19:59:43.313410 29714 raft_consensus.cc:491] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
W20250905 19:59:43.382583 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 166119us
W20250905 19:59:43.356957 29713 consensus_meta.cc:287] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.709s user 0.009s sys 0.001s
W20250905 19:59:43.422623 28005 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.919s user 0.138s sys 0.189s
I20250905 19:59:43.406370 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:43.419379 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 19:59:43.423835 28125 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.757s user 0.016s sys 0.029s
W20250905 19:59:43.430354 28125 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.774s user 0.017s sys 0.029s
W20250905 19:59:43.430691 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 780246us
I20250905 19:59:43.381626 29813 consensus_queue.cc:237] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:43.425966 28123 leader_election.cc:400] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Vote denied by peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299) with higher term. Message: Invalid argument: T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 3. Current term is 4.
I20250905 19:59:43.433303 28123 leader_election.cc:403] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20250905 19:59:43.427448 27973 leader_election.cc:304] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:43.430056 28080 raft_consensus.cc:2466] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:43.433481 28752 raft_consensus.cc:2466] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:59:43.428400 28005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1234) took 1943 ms. Trace:
I20250905 19:59:43.435007 28077 raft_consensus.cc:2466] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:43.435163 29946 raft_consensus.cc:491] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
W20250905 19:59:43.422463 28716 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.933s user 0.007s sys 0.007s
I20250905 19:59:43.435129 27973 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:43.432053 28125 leader_election.cc:304] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:43.382936 29869 leader_election.cc:290] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:43.185546 29930 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:43.386502 29941 raft_consensus.cc:491] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:43.434590 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:43.439333 27973 leader_election.cc:304] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:43.443998 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 19:59:43.444599 29800 consensus_meta.cc:287] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.842s user 0.010s sys 0.000s
I20250905 19:59:43.444893 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:43.445426 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:43.449594 28049 leader_election.cc:304] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:43.454119 27973 leader_election.cc:304] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:43.434813 28005 rpcz_store.cc:276] 0905 19:59:41.484945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:41.731730 (+246785us) service_pool.cc:225] Handling call
0905 19:59:41.732216 (+ 486us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:41.732468 (+ 252us) raft_consensus.cc:1532] Early marking committed up to index 3
0905 19:59:42.502981 (+770513us) log.cc:844] Serialized 14 byte log entry
0905 19:59:43.428019 (+925038us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:43.428084 (+ 65us) raft_consensus.cc:1643] Marking committed up to 3
0905 19:59:43.428138 (+ 54us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:43.428217 (+ 79us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:43.428344 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15394,"prepare.queue_time_us":504771,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":9068,"raft.queue_time_us":6223,"thread_start_us":1694472,"threads_started":2,"wal-append.queue_time_us":921169}
I20250905 19:59:43.431856 29737 raft_consensus.cc:2747] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:43.462209 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:43.457242 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:43.436416 29895 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:43.446797 29622 consensus_meta.cc:287] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.726s user 0.010s sys 0.000s
W20250905 19:59:43.456972 29848 consensus_meta.cc:287] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.642s user 0.010s sys 0.000s
I20250905 19:59:43.438400 29599 raft_consensus.cc:3058] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:43.459721 29800 raft_consensus.cc:513] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:43.477245 28080 raft_consensus.cc:2391] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 3: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:43.472368 28005 raft_consensus.cc:2466] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
I20250905 19:59:43.444005 29849 leader_election.cc:290] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:43.436252 29866 leader_election.cc:290] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:43.452093 29809 raft_consensus.cc:491] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:43.276770 28748 rpcz_store.cc:276] 0905 19:59:41.299680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:41.302761 (+ 3081us) service_pool.cc:225] Handling call
0905 19:59:41.334103 (+ 31342us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:41.775770 (+441667us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:41.779855 (+ 4085us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:41.840346 (+ 60491us) raft_consensus.cc:1643] Marking committed up to 4
0905 19:59:41.843723 (+ 3377us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:42.955406 (+1111683us) spinlock_profiling.cc:243] Waited 363 ms on lock 0x7b54006103d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:43.050655 (+ 95249us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:43.100785 (+ 50130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":834881280}
I20250905 19:59:43.425513 29945 raft_consensus.cc:491] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:43.482663 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:43.440246 29876 raft_consensus.cc:2747] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:43.438560 29931 leader_election.cc:290] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:43.489076 29815 consensus_meta.cc:287] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.949s user 0.008s sys 0.002s
I20250905 19:59:43.490154 29943 raft_consensus.cc:2802] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:43.474493 29622 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:43.523928 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:43.525059 28005 raft_consensus.cc:2366] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 5. Current term is 6.
I20250905 19:59:43.441565 29881 raft_consensus.cc:2802] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:43.495308 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:43.495355 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 19:59:43.473927 29923 log.cc:927] Time spent T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.333s user 0.001s sys 0.000s
I20250905 19:59:43.530033 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:43.530927 28080 raft_consensus.cc:2466] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 19:59:43.420395 29745 raft_consensus.cc:513] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:43.460654 29903 raft_consensus.cc:2802] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250905 19:59:43.499351 28154 consensus_meta.cc:287] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.816s user 0.015s sys 0.017s
I20250905 19:59:43.538740 28154 raft_consensus.cc:2466] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
I20250905 19:59:43.433620 29713 raft_consensus.cc:513] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:43.501919 28006 raft_consensus.cc:2466] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:43.542465 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:43.543349 28006 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:43.433023 29714 raft_consensus.cc:513] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:43.438165 29934 raft_consensus.cc:491] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:43.491045 29869 raft_consensus.cc:491] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:43.509359 29800 leader_election.cc:290] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:43.474377 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 19:59:43.494499 29756 consensus_meta.cc:287] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.764s user 0.017s sys 0.008s
I20250905 19:59:43.451998 29906 leader_election.cc:290] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:43.451845 29933 raft_consensus.cc:2802] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:43.519707 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:43.532136 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:43.533525 27973 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:43.460650 29941 raft_consensus.cc:513] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:43.537206 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:43.539484 28154 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 1887) took 1033 ms. Trace:
W20250905 19:59:43.593931 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 60548.1us
I20250905 19:59:43.459971 29937 consensus_queue.cc:1035] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
I20250905 19:59:43.587466 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 19:59:43.545679 29788 consensus_meta.cc:287] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.283s user 0.010s sys 0.006s
I20250905 19:59:43.535900 29881 raft_consensus.cc:491] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:43.441462 29946 raft_consensus.cc:513] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:43.490643 29895 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:43.506009 29809 raft_consensus.cc:513] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:43.537554 29875 leader_election.cc:290] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:43.589725 28125 leader_election.cc:400] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Vote denied by peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421) with higher term. Message: Invalid argument: T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3.
I20250905 19:59:43.615309 28125 leader_election.cc:403] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250905 19:59:43.553018 29511 leader_election.cc:290] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:43.607070 29881 raft_consensus.cc:3058] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:43.558605 29869 raft_consensus.cc:513] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:43.599756 28005 raft_consensus.cc:2408] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250905 19:59:43.594825 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:43.627213 28080 raft_consensus.cc:2391] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 3: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:43.628265 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:43.594815 28154 rpcz_store.cc:276] 0905 19:59:42.506248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:42.506394 (+ 146us) service_pool.cc:225] Handling call
0905 19:59:43.539398 (+1033004us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:43.554095 29878 consensus_meta.cc:287] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 1.352s user 0.014s sys 0.001s
I20250905 19:59:43.599062 27973 leader_election.cc:304] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:43.635828 28080 raft_consensus.cc:2391] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 4: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
W20250905 19:59:43.557310 29854 consensus_meta.cc:287] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.955s user 0.016s sys 0.001s
I20250905 19:59:43.639585 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:59:43.556380 29905 consensus_meta.cc:287] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.896s user 0.005s sys 0.008s
I20250905 19:59:43.516008 29945 raft_consensus.cc:513] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:43.606650 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:43.589507 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:43.527509 29815 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:43.545385 29943 raft_consensus.cc:491] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:43.606626 28748 raft_consensus.cc:2466] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 19:59:43.588491 29622 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:43.591640 29745 leader_election.cc:290] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:43.591174 29787 raft_consensus.cc:513] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:43.602574 29788 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:43.662222 28157 raft_consensus.cc:2391] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:43.557230 29824 leader_election.cc:290] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:43.557114 29873 leader_election.cc:290] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:43.555369 29867 leader_election.cc:290] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:43.587764 29939 raft_consensus.cc:2802] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:43.597302 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:43.566907 29800 raft_consensus.cc:491] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:43.633221 28005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1839) took 1016 ms. Trace:
W20250905 19:59:43.633512 28009 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 178) took 1245 ms (client timeout 1620 ms). Trace:
I20250905 19:59:43.546077 29903 raft_consensus.cc:491] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:43.640465 28154 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 1895) took 1044 ms. Trace:
I20250905 19:59:43.671247 28154 rpcz_store.cc:276] 0905 19:59:42.596658 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:43.639286 (+1042628us) service_pool.cc:225] Handling call
0905 19:59:43.640406 (+ 1120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:43.640939 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:43.632742 29936 raft_consensus.cc:491] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:43.523674 29883 consensus_queue.cc:260] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:43.612253 29933 raft_consensus.cc:695] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:43.623440 29810 raft_consensus.cc:3058] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
W20250905 19:59:43.635432 28745 consensus_meta.cc:287] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.908s user 0.011s sys 0.000s
I20250905 19:59:43.569134 29756 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:43.620767 29508 leader_election.cc:290] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:43.602686 29906 raft_consensus.cc:2802] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 19:59:43.593828 29934 raft_consensus.cc:513] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:43.643815 29854 raft_consensus.cc:513] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:43.660398 29941 leader_election.cc:290] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:43.691642 28744 raft_consensus.cc:2366] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6.
I20250905 19:59:43.660960 29787 leader_election.cc:290] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:43.623715 29918 leader_election.cc:290] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:43.662889 28159 raft_consensus.cc:2366] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 6. Current term is 7.
I20250905 19:59:43.668128 28123 leader_election.cc:304] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 19:59:43.668682 28005 rpcz_store.cc:276] 0905 19:59:42.616827 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:43.536786 (+919959us) service_pool.cc:225] Handling call
0905 19:59:43.633142 (+ 96356us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:43.667897 28009 rpcz_store.cc:269] 0905 19:59:42.388041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:42.503841 (+115800us) service_pool.cc:225] Handling call
0905 19:59:43.633350 (+1129509us) spinlock_profiling.cc:243] Waited 237 ms on lock 0x7b54003831d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:43.633460 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":544505088}
I20250905 19:59:43.645432 29876 raft_consensus.cc:491] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:43.672164 28080 raft_consensus.cc:2466] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:59:43.673353 29945 leader_election.cc:290] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:43.652375 29943 raft_consensus.cc:3058] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:43.679555 28155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 1899) took 1019 ms. Trace:
W20250905 19:59:43.677456 28125 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:54164 dropped due to backpressure: service queue is full with 50 items [suppressed 19 similar messages]
I20250905 19:59:43.674027 29936 raft_consensus.cc:513] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:43.677691 28007 raft_consensus.cc:2466] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:43.686578 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:43.693789 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:43.667002 29894 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:43.674676 29903 raft_consensus.cc:3058] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:43.735199 28083 raft_consensus.cc:2374] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2. Re-sending same reply.
I20250905 19:59:43.736673 28007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1237) took 1347 ms. Trace:
I20250905 19:59:43.737082 28007 rpcz_store.cc:276] 0905 19:59:42.389278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:43.341864 (+952586us) service_pool.cc:225] Handling call
0905 19:59:43.736516 (+394652us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:43.687101 28743 raft_consensus.cc:2466] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 19:59:43.656714 29924 raft_consensus.cc:2747] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 19:59:43.697139 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:43.682771 29930 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:43.648129 29905 raft_consensus.cc:513] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:43.698622 29906 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250905 19:59:43.732384 29805 consensus_meta.cc:287] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.138s user 0.002s sys 0.024s
I20250905 19:59:43.681349 29939 raft_consensus.cc:491] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:43.674846 29800 raft_consensus.cc:513] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:43.660766 27975 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:54164 dropped due to backpressure: service queue is full with 50 items
I20250905 19:59:43.676446 29809 leader_election.cc:290] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:43.706918 28001 raft_consensus.cc:2391] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 6: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:43.711079 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:43.711251 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:43.700004 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:43.727958 29815 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:43.728695 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:43.728394 29850 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:43.732457 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:43.733434 28152 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33752 (request call id 1411) took 2576 ms (client timeout 1844 ms). Trace:
W20250905 19:59:43.800899 28152 rpcz_store.cc:269] 0905 19:59:41.156615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:41.161840 (+ 5225us) service_pool.cc:225] Handling call
0905 19:59:43.733216 (+2571376us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b5400186854. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:43.733338 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3505329920}
W20250905 19:59:43.801916 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:43.734186 28004 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 172) took 2577 ms (client timeout 1552 ms). Trace:
I20250905 19:59:43.691923 28745 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
I20250905 19:59:43.738029 28155 rpcz_store.cc:276] 0905 19:59:42.660531 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:42.791311 (+130780us) service_pool.cc:225] Handling call
0905 19:59:43.679349 (+888038us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x7b54000ed3d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:43.679489 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":520863616}
I20250905 19:59:43.692155 29883 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:43.741547 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:43.803335 28083 raft_consensus.cc:2466] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:43.747251 29941 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:43.742018 28154 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 1896) took 1141 ms. Trace:
I20250905 19:59:43.661710 29946 leader_election.cc:290] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:43.748781 28050 leader_election.cc:304] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:43.728459 29934 leader_election.cc:290] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:43.715128 29876 raft_consensus.cc:513] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:43.771442 29939 raft_consensus.cc:3058] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:43.772323 29756 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:43.726701 29873 raft_consensus.cc:2802] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:43.816681 29873 raft_consensus.cc:491] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:43.782258 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:43.819658 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:43.819830 28750 raft_consensus.cc:2466] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 19:59:43.820282 28002 raft_consensus.cc:2391] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:43.820468 28155 raft_consensus.cc:2391] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 4: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:43.714884 29713 leader_election.cc:290] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:43.736411 29714 leader_election.cc:290] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:43.786240 28080 raft_consensus.cc:3058] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
W20250905 19:59:43.773809 29888 consensus_meta.cc:287] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 1.049s user 0.010s sys 0.003s
I20250905 19:59:43.773563 29905 leader_election.cc:290] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:43.643939 29878 raft_consensus.cc:513] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:43.747522 29787 raft_consensus.cc:491] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:43.802474 28004 rpcz_store.cc:269] 0905 19:59:41.156807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:41.157011 (+ 204us) service_pool.cc:225] Handling call
0905 19:59:43.733947 (+2576936us) spinlock_profiling.cc:243] Waited 1.94 s on lock 0x7b540016dfd4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:43.734089 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4451576704}
I20250905 19:59:43.805965 28154 rpcz_store.cc:276] 0905 19:59:42.600511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:43.680111 (+1079600us) service_pool.cc:225] Handling call
0905 19:59:43.741957 (+ 61846us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:43.809216 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 60541.1us
I20250905 19:59:43.835551 28050 leader_election.cc:304] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:43.810050 29864 raft_consensus.cc:2802] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election won for term 3
W20250905 19:59:43.811211 28749 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.055s user 0.040s sys 0.028s
W20250905 19:59:43.844925 28749 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.088s user 0.040s sys 0.028s
I20250905 19:59:43.845566 28050 leader_election.cc:304] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:43.845994 28749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40904 (request call id 491) took 1117 ms. Trace:
I20250905 19:59:43.811928 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:43.813124 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:43.847179 28050 leader_election.cc:304] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:43.750185 29906 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Advancing to term 7
I20250905 19:59:43.847631 28152 raft_consensus.cc:2466] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
I20250905 19:59:43.786164 29788 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:43.828270 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:43.808678 28745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40904 (request call id 486) took 1383 ms. Trace:
I20250905 19:59:43.683097 27973 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:54164 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1853)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1863)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1861)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1247)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 183)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1860)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1874)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1868)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1875)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1249)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 185)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1871)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 184)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1876)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1872)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 180)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1241)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1244)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1252)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1873)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1877)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1856)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1245)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1878)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1869)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1246)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1866)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 187)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1870)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 182)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1256)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 181)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1862)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1865)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1867)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1851)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1238)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1843)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1847)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1855)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1859)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1248)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1864)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 186)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1253)
I20250905 19:59:43.782913 29805 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:43.699126 29933 consensus_queue.cc:237] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:43.819475 29871 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:43.815367 29869 leader_election.cc:290] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:43.843509 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:43.811018 29941 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:43.847529 28083 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 19:59:43.788036 29895 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:43.822500 28123 leader_election.cc:304] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 19:59:43.847859 29840 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:43.846581 28749 rpcz_store.cc:276] 0905 19:59:42.728342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:42.729135 (+ 793us) service_pool.cc:225] Handling call
0905 19:59:42.729464 (+ 329us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:42.729781 (+ 317us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:42.729806 (+ 25us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:42.730302 (+ 496us) log.cc:844] Serialized 26 byte log entry
0905 19:59:43.845466 (+1115164us) raft_consensus.cc:1643] Marking committed up to 5
0905 19:59:43.845530 (+ 64us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:43.845645 (+ 115us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:43.845711 (+ 66us) raft_consensus.cc:1683] finished
0905 19:59:43.845731 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:43.845920 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":58917,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":9416,"thread_start_us":1088722,"threads_started":1,"wal-append.queue_time_us":1028850}
I20250905 19:59:43.848978 28050 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:43.823858 29873 raft_consensus.cc:3058] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:43.852984 28081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1352) took 1007 ms. Trace:
I20250905 19:59:43.944867 29788 raft_consensus.cc:491] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
W20250905 19:59:43.981618 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 304163us
I20250905 19:59:43.981556 28081 rpcz_store.cc:276] 0905 19:59:42.845253 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:43.807800 (+962547us) service_pool.cc:225] Handling call
0905 19:59:43.852921 (+ 45121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:43.854615 28084 raft_consensus.cc:2466] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:59:43.859076 28750 raft_consensus.cc:2391] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 19:59:43.843868 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:43.860652 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:43.851495 29508 raft_consensus.cc:2802] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250905 19:59:43.896262 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 47376.7us
I20250905 19:59:43.985545 28122 leader_election.cc:304] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:43.898763 28718 leader_election.cc:304] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:43.853266 29644 raft_consensus.cc:2802] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:43.902503 27973 leader_election.cc:304] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 19:59:43.987890 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 85521.6us
I20250905 19:59:43.988761 27973 leader_election.cc:304] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:43.831043 29888 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:43.945451 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:43.948561 29854 leader_election.cc:290] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:43.860009 29864 raft_consensus.cc:695] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:43.851879 29787 raft_consensus.cc:513] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:43.852687 28745 rpcz_store.cc:276] 0905 19:59:42.425211 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:42.429260 (+ 4049us) service_pool.cc:225] Handling call
0905 19:59:43.806851 (+1377591us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:43.895394 29901 raft_consensus.cc:2802] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 19:59:43.944248 29826 raft_consensus.cc:2747] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250905 19:59:43.858096 29871 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:43.972841 29936 leader_election.cc:290] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:43.869150 28151 raft_consensus.cc:2391] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 4: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:43.984643 28050 leader_election.cc:304] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:44.001259 29924 raft_consensus.cc:2802] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:44.001080 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 19:59:44.001116 29948 log.cc:927] Time spent T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.226s user 0.000s sys 0.001s
I20250905 19:59:44.004225 28125 leader_election.cc:304] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:44.005012 29930 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:44.013877 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 115246us
I20250905 19:59:43.993366 29800 leader_election.cc:290] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:44.008433 29756 raft_consensus.cc:2802] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:44.009733 29934 raft_consensus.cc:2802] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:44.016448 27974 leader_election.cc:304] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:44.022387 29928 raft_consensus.cc:491] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:44.008103 29876 leader_election.cc:290] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:44.035706 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:43.947084 29840 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:44.039841 28083 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
I20250905 19:59:44.058130 28083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1344) took 1292 ms. Trace:
I20250905 19:59:44.058614 28083 rpcz_store.cc:276] 0905 19:59:42.765731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:43.807464 (+1041733us) service_pool.cc:225] Handling call
0905 19:59:44.058045 (+250581us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:44.040294 29871 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:44.036242 27973 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:54164 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1860)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1874)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1868)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1875)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1249)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 185)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1871)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 184)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1876)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1872)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 180)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1241)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1244)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1252)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1873)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1877)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1856)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1245)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1878)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1869)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 188)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1246)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1866)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 189)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 187)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1870)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 182)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1256)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1259)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 181)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 190)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1862)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1865)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1867)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1851)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1238)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1843)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1847)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1855)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1859)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1248)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1864)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 186)
I20250905 19:59:44.063208 27974 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:44.062268 27973 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:45844 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1860)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1874)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1868)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1875)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1249)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 185)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1871)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 184)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1876)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1872)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 180)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1241)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1244)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1252)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1873)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1877)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1856)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1245)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1878)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1869)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 188)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1246)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1866)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 189)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 187)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1870)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 182)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1256)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1259)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 181)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 190)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1862)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1865)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1867)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1851)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1238)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1843)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1847)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1855)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1859)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1248)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1864)
W20250905 19:59:44.020135 28077 consensus_meta.cc:287] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.538s user 0.011s sys 0.000s
I20250905 19:59:44.029446 29644 raft_consensus.cc:695] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 2 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
W20250905 19:59:44.012434 29904 consensus_meta.cc:287] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.141s user 0.014s sys 0.000s
W20250905 19:59:44.043038 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 58501.2us
W20250905 19:59:44.019734 29853 consensus_meta.cc:287] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 1.421s user 0.011s sys 0.003s
I20250905 19:59:44.042676 28008 raft_consensus.cc:2391] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 4: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:44.046808 28122 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:43.990614 29508 raft_consensus.cc:491] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:44.051563 28155 raft_consensus.cc:2466] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:44.014591 29801 raft_consensus.cc:491] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:44.023947 29894 raft_consensus.cc:2802] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250905 19:59:44.060858 29957 log.cc:927] Time spent T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.250s user 0.000s sys 0.001s
I20250905 19:59:44.007963 29873 raft_consensus.cc:513] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:44.067626 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:44.052306 29964 raft_consensus.cc:491] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:44.053718 29756 raft_consensus.cc:491] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:44.109319 28083 raft_consensus.cc:2466] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 1.
I20250905 19:59:44.060246 29864 consensus_queue.cc:237] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:44.081110 29880 leader_election.cc:290] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:44.083423 29805 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 19:59:44.084977 29818 consensus_meta.cc:287] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.843s user 0.013s sys 0.000s
W20250905 19:59:44.085551 29505 consensus_meta.cc:287] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.679s user 0.008s sys 0.000s
I20250905 19:59:44.057168 29854 raft_consensus.cc:2802] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:44.044037 29850 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:44.097632 29904 raft_consensus.cc:513] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:44.020922 29788 raft_consensus.cc:513] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:44.129046 29854 raft_consensus.cc:695] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:44.129987 29854 consensus_queue.cc:237] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:44.130470 29904 leader_election.cc:290] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:44.130633 29788 leader_election.cc:290] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:44.093766 28050 leader_election.cc:304] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:44.105649 29801 raft_consensus.cc:513] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:44.131755 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 53321.8us
I20250905 19:59:44.131978 29833 raft_consensus.cc:2802] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:44.132413 29833 raft_consensus.cc:491] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:44.132823 29833 raft_consensus.cc:3058] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:44.094446 27973 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:54156 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1860)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1874)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1868)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1875)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1249)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 185)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1871)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 184)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1876)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1872)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 180)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1241)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1244)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1252)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1873)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1877)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1856)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1245)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1878)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1869)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 188)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1246)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1866)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 189)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 187)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1870)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 182)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1256)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1259)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 181)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 190)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1862)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1865)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1867)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1851)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1238)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1843)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1847)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1855)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1859)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1248)
I20250905 19:59:44.097889 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 19:59:44.133558 29956 log.cc:927] Time spent T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.175s user 0.000s sys 0.000s
I20250905 19:59:44.133744 28745 raft_consensus.cc:2466] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:59:44.134701 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:44.135432 28085 raft_consensus.cc:2391] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 2: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:44.135787 28085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1345) took 1367 ms. Trace:
I20250905 19:59:44.136078 28085 rpcz_store.cc:276] 0905 19:59:42.768564 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:42.770112 (+ 1548us) service_pool.cc:225] Handling call
0905 19:59:44.134559 (+1364447us) spinlock_profiling.cc:243] Waited 6.33 ms on lock 0x7b5800204f98. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac671cc 00007fe49ac66d8b 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b7db60 00007fe4a2b7d444 00007fe49c2050c4 00007fe49c205052
0905 19:59:44.135736 (+ 1177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14701696}
I20250905 19:59:44.114771 29756 raft_consensus.cc:3058] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:44.137115 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:44.137876 28085 raft_consensus.cc:2391] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 6: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:44.028750 29901 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:44.096611 28081 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 3.
I20250905 19:59:44.139283 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:44.138998 29644 consensus_queue.cc:237] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:44.092098 29508 raft_consensus.cc:3058] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:44.100210 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:44.141934 28155 raft_consensus.cc:2466] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
W20250905 19:59:44.142020 28001 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1840) took 1525 ms (client timeout 1594 ms). Trace:
W20250905 19:59:44.142390 28001 rpcz_store.cc:269] 0905 19:59:42.617056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:42.758238 (+141182us) service_pool.cc:225] Handling call
0905 19:59:44.141820 (+1383582us) spinlock_profiling.cc:243] Waited 866 ms on lock 0x7b540038fc54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:44.141960 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1991271680}
I20250905 19:59:44.142834 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:44.049701 29826 raft_consensus.cc:2802] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:44.104935 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:44.143432 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:44.143800 28747 raft_consensus.cc:3058] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:44.144025 28001 raft_consensus.cc:3058] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:44.099761 29884 raft_consensus.cc:2802] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election won for term 6
I20250905 19:59:44.054996 29924 raft_consensus.cc:695] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:44.149138 29884 raft_consensus.cc:695] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:44.097543 29894 raft_consensus.cc:491] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:44.150010 29924 consensus_queue.cc:237] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:44.065569 29928 raft_consensus.cc:513] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:44.119109 28715 leader_election.cc:304] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 19:59:44.176097 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 57051.5us
I20250905 19:59:44.121656 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:44.123401 28749 raft_consensus.cc:1273] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250905 19:59:44.098184 29853 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:44.181459 28001 raft_consensus.cc:2466] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
W20250905 19:59:44.128315 29908 log.cc:927] Time spent T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 1.876s user 0.005s sys 0.001s
I20250905 19:59:44.182531 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:44.182680 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:44.183346 28082 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1324) took 3332 ms. Trace:
I20250905 19:59:44.126520 29878 leader_election.cc:290] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:44.134141 27973 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:54164 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1860)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1874)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1868)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1875)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1249)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 185)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1871)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 184)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1876)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1872)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 180)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1241)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1244)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1252)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1873)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1877)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1856)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1245)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1878)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1869)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 188)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1246)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1866)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 189)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 187)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1870)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 182)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1256)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1259)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 181)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 190)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1862)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1865)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1867)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1851)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1238)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1843)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1847)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1855)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1859)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1248)
I20250905 19:59:44.066550 29930 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:44.139881 28081 raft_consensus.cc:3058] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:44.100346 29810 raft_consensus.cc:2747] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421) with higher term. Message: Invalid argument: T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3.
I20250905 19:59:44.098557 29905 raft_consensus.cc:2802] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:44.146939 28153 raft_consensus.cc:1238] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 2. Current term is 3. Ops: []
W20250905 19:59:44.147820 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:44.074290 28077 raft_consensus.cc:2466] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:44.138222 29901 raft_consensus.cc:3058] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:44.150838 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:44.053911 29934 raft_consensus.cc:491] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
W20250905 19:59:44.112448 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 65747.7us
I20250905 19:59:44.154870 28050 leader_election.cc:304] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:44.197988 28122 leader_election.cc:304] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:44.117513 28080 raft_consensus.cc:2466] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:44.119045 29871 raft_consensus.cc:2747] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250905 19:59:44.166005 29894 raft_consensus.cc:3058] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:44.153184 29818 raft_consensus.cc:513] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:44.172518 29900 consensus_meta.cc:287] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 1.500s user 0.014s sys 0.014s
I20250905 19:59:44.183658 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:44.183724 28082 rpcz_store.cc:276] 0905 19:59:40.850991 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:41.315090 (+464099us) service_pool.cc:225] Handling call
0905 19:59:41.334746 (+ 19656us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:41.392573 (+ 57827us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:41.392602 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:41.400277 (+ 7675us) log.cc:844] Serialized 26 byte log entry
0905 19:59:42.227190 (+826913us) raft_consensus.cc:1643] Marking committed up to 1
0905 19:59:42.227251 (+ 61us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:42.227345 (+ 94us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:44.109644 (+1882299us) spinlock_profiling.cc:243] Waited 42.1 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49d85891e 00007fe49d857c66 00007fe49d857888 00007fe49d8612c1 00007fe49d861279 00007fe49d861209
0905 19:59:44.183076 (+ 73432us) raft_consensus.cc:1683] finished
0905 19:59:44.183097 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:44.183289 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":57420,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":14417,"spinlock_wait_cycles":96805120,"thread_start_us":756297,"threads_started":1,"wal-append.queue_time_us":831796}
I20250905 19:59:44.132982 29801 leader_election.cc:290] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:44.229230 28752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40918 (request call id 431) took 1495 ms. Trace:
I20250905 19:59:44.188558 29936 raft_consensus.cc:2802] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:44.235364 28752 rpcz_store.cc:276] 0905 19:59:42.734134 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:42.734358 (+ 224us) service_pool.cc:225] Handling call
0905 19:59:44.216894 (+1482536us) spinlock_profiling.cc:243] Waited 670 ms on lock 0x7b54005302d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:44.229158 (+ 12264us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1540972416}
I20250905 19:59:44.194666 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:44.108877 29906 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:44.181903 29853 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:44.143577 29826 raft_consensus.cc:491] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:44.212329 29950 raft_consensus.cc:2802] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:44.249783 29826 raft_consensus.cc:3058] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:44.150878 29505 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:44.204308 29934 raft_consensus.cc:3058] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:44.204715 28081 raft_consensus.cc:2466] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:44.201930 28122 leader_election.cc:304] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 19:59:44.257901 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 56081.2us
I20250905 19:59:44.259624 28122 leader_election.cc:304] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:44.207998 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:44.175033 29941 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:44.309906 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:44.310789 28083 raft_consensus.cc:2466] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 19:59:44.217628 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:44.229839 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:44.312791 28079 raft_consensus.cc:2466] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:44.313349 28079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1336) took 1667 ms. Trace:
I20250905 19:59:44.219271 29931 raft_consensus.cc:2802] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:44.241128 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:44.211525 27973 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:54164 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1875)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1249)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 185)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1871)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 184)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1876)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1872)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 180)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1241)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1244)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1252)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1873)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1877)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1856)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1245)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1878)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1869)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 188)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1246)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1866)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 189)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 187)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1870)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 182)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1256)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1259)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 181)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1263)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 190)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1862)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1266)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1865)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1867)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1851)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1238)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1843)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1847)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1855)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1859)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1248)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1264)
I20250905 19:59:44.243944 28155 raft_consensus.cc:2391] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 4: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:44.267524 28003 raft_consensus.cc:2366] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 2. Current term is 3.
I20250905 19:59:44.273447 28122 leader_election.cc:304] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 19:59:44.335069 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 61706.7us
W20250905 19:59:44.335990 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:44.336740 28122 leader_election.cc:304] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 19:59:44.337631 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:44.150153 29884 consensus_queue.cc:237] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:44.338838 29805 raft_consensus.cc:2802] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:44.339622 29805 raft_consensus.cc:695] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:44.338876 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:44.341496 28752 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 19:59:44.263885 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:44.347960 28081 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 7
I20250905 19:59:44.237169 28049 leader_election.cc:304] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:44.310101 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:44.313699 28079 rpcz_store.cc:276] 0905 19:59:42.646261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:42.785435 (+139174us) service_pool.cc:225] Handling call
0905 19:59:44.216965 (+1431530us) spinlock_profiling.cc:243] Waited 5.79 ms on lock 0x7b5800204f98. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b7db60 00007fe4a2b7d444 00007fe49c2050c4 00007fe49c205052
0905 19:59:44.313281 (+ 96316us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13328256}
I20250905 19:59:44.351747 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:44.352124 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:44.352561 28079 raft_consensus.cc:2391] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 5: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:44.352953 28001 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:44.314826 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:44.315819 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:44.248924 29645 consensus_meta.cc:287] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.285s user 0.016s sys 0.004s
I20250905 19:59:44.210692 29905 raft_consensus.cc:695] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:44.353098 28079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1324) took 1087 ms. Trace:
I20250905 19:59:44.357882 28079 rpcz_store.cc:276] 0905 19:59:43.265373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.351390 (+1086017us) service_pool.cc:225] Handling call
0905 19:59:44.353040 (+ 1650us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:44.209549 29849 raft_consensus.cc:2747] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 19:59:44.243402 29936 raft_consensus.cc:491] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:44.360242 28152 raft_consensus.cc:2391] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 6: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:44.359211 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:44.360760 28080 raft_consensus.cc:3058] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:44.363397 28158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:33742 (request call id 637) took 1847 ms. Trace:
I20250905 19:59:44.214164 29930 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:44.363770 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:44.312830 29848 raft_consensus.cc:2802] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:44.364384 28083 raft_consensus.cc:3058] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
W20250905 19:59:44.361876 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 124817us
I20250905 19:59:44.313175 29871 raft_consensus.cc:2802] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:44.214716 29939 raft_consensus.cc:513] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:44.216437 29955 consensus_queue.cc:1035] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.158s
I20250905 19:59:44.315321 29737 raft_consensus.cc:491] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:44.367581 29941 raft_consensus.cc:2802] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:44.403733 29787 leader_election.cc:290] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:44.409724 29894 raft_consensus.cc:513] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:44.375934 29818 leader_election.cc:290] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:43.775317 27975 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:54164 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 183)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1860)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1874)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1868)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1875)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1249)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 185)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1871)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 184)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1876)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1872)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 180)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1241)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1244)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1252)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1873)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1877)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1856)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1245)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1878)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1869)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1246)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1866)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 187)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1870)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 182)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1256)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 181)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1862)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1865)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1867)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1851)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1238)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1843)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1847)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1855)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1859)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1248)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1864)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 186)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1253)
I20250905 19:59:44.407825 29941 raft_consensus.cc:491] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:44.421285 29941 raft_consensus.cc:3058] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
W20250905 19:59:44.421492 28157 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 1904) took 1606 ms (client timeout 1590 ms). Trace:
I20250905 19:59:44.421991 29505 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:44.412364 29737 raft_consensus.cc:513] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:44.424160 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:44.421928 27975 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:54164 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 185)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1871)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 184)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1876)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1872)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 180)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1241)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1244)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1252)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1873)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1877)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1856)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1245)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1878)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1869)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 188)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1246)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1866)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 189)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 187)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1870)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 182)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1256)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1259)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 181)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1263)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 190)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1862)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1879)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1269)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1270)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1266)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1865)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1867)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1851)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1238)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1843)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1847)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1855)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1859)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1248)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1264)
I20250905 19:59:44.422343 28150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 1890) took 1911 ms. Trace:
I20250905 19:59:44.426029 28150 rpcz_store.cc:276] 0905 19:59:42.510859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:42.513420 (+ 2561us) service_pool.cc:225] Handling call
0905 19:59:44.422160 (+1908740us) spinlock_profiling.cc:243] Waited 719 ms on lock 0x7b5400315e54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:44.422288 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1653002624}
W20250905 19:59:44.421938 28157 rpcz_store.cc:269] 0905 19:59:42.815475 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:43.349365 (+533890us) service_pool.cc:225] Handling call
0905 19:59:44.421283 (+1071918us) spinlock_profiling.cc:243] Waited 312 ms on lock 0x7b540037f0d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:44.421422 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":717300992}
W20250905 19:59:44.403923 29929 log.cc:927] Time spent T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20250905 19:59:44.341040 29888 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:44.378648 29871 raft_consensus.cc:695] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:44.381044 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:44.378676 29873 leader_election.cc:290] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:44.363981 29936 raft_consensus.cc:3058] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:44.367094 29809 raft_consensus.cc:2802] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 19:59:44.322008 29931 raft_consensus.cc:491] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:44.385535 28083 raft_consensus.cc:2466] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 3.
I20250905 19:59:44.385689 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:44.363811 28158 rpcz_store.cc:276] 0905 19:59:42.508688 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:42.509800 (+ 1112us) service_pool.cc:225] Handling call
0905 19:59:44.343761 (+1833961us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b5400073ad4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1f6e5 00007fe49dc179ba 00007fe4a2b7e089 00007fe49c207644 00007fe49c2075d2 00007fe49c207501 00007fe49c20747c 00007fe49c206732 00007fe49c018ddc 00007fe49c018216
0905 19:59:44.355626 (+ 11865us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2919608576}
I20250905 19:59:44.455799 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:59:44.455971 27894 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet e707b0f561934674947fb463ce626a07 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 26b2f874b8e54f08b01a9ed00d5cce05 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:44.456667 28158 raft_consensus.cc:2466] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 1.
I20250905 19:59:44.379926 27973 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:54164 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 185)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1871)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 184)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1876)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1872)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 180)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1241)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1244)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1252)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1873)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1877)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1856)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1245)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1878)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1869)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 188)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1246)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1866)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 189)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 187)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1870)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 182)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1256)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1259)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 181)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1263)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 190)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1862)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1269)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1270)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1266)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1865)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1867)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1851)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1238)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1843)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1847)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1855)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1859)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1248)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1264)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1267)
I20250905 19:59:44.384251 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:44.388895 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:44.169198 29964 raft_consensus.cc:513] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:44.363767 29645 raft_consensus.cc:513] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:44.397250 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:44.399485 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:44.242882 29900 raft_consensus.cc:513] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:44.414605 28076 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1332) took 1773 ms (client timeout 1912 ms). Trace:
I20250905 19:59:44.398305 29508 raft_consensus.cc:513] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:44.416031 29599 consensus_meta.cc:287] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.920s user 0.011s sys 0.000s
W20250905 19:59:44.416661 29881 consensus_meta.cc:287] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.798s user 0.011s sys 0.000s
I20250905 19:59:44.383898 29905 consensus_queue.cc:237] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:44.377199 29848 raft_consensus.cc:695] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
W20250905 19:59:44.433145 28159 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 1892) took 1921 ms (client timeout 1611 ms). Trace:
W20250905 19:59:44.433795 28151 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 1898) took 1803 ms (client timeout 1545 ms). Trace:
I20250905 19:59:44.429592 27975 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:54164 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 185)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1871)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 184)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1876)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1872)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 180)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1241)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1244)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1252)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1873)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1877)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1856)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1245)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1878)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1869)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 188)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1246)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1866)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 189)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 187)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1870)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 182)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1256)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1259)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 181)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1263)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 190)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1862)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1879)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1880)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1269)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1270)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1266)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1865)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1867)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1851)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1238)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1843)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1847)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1855)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1859)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1248)
I20250905 19:59:44.434309 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:59:44.486238 28151 rpcz_store.cc:269] 0905 19:59:42.630111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:42.788241 (+158130us) service_pool.cc:225] Handling call
0905 19:59:44.433603 (+1645362us) spinlock_profiling.cc:243] Waited 995 ms on lock 0x7b5400073ad4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:44.433732 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2288998144}
I20250905 19:59:44.435424 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:44.487670 28157 raft_consensus.cc:2466] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 1.
I20250905 19:59:44.436332 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:44.488670 28156 raft_consensus.cc:2366] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4.
W20250905 19:59:44.442066 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 286716us
I20250905 19:59:44.462854 29931 raft_consensus.cc:3058] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:44.461849 28082 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1354) took 1602 ms. Trace:
I20250905 19:59:44.495224 28082 rpcz_store.cc:276] 0905 19:59:42.859337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.360477 (+1501140us) service_pool.cc:225] Handling call
0905 19:59:44.461760 (+101283us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:44.463416 28752 raft_consensus.cc:3058] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:44.464717 28715 leader_election.cc:304] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:44.497427 27975 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1248) took 1018 ms. Trace:
I20250905 19:59:44.415419 29894 leader_election.cc:290] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:44.468371 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:44.469709 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:44.453691 29809 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250905 19:59:44.471405 28078 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1321) took 1423 ms (client timeout 1819 ms). Trace:
I20250905 19:59:44.497848 27975 rpcz_store.cc:276] 0905 19:59:43.478584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.497377 (+1018793us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250905 19:59:44.505849 28078 rpcz_store.cc:269] 0905 19:59:43.048222 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.396943 (+1348721us) service_pool.cc:225] Handling call
0905 19:59:44.471314 (+ 74371us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:44.481774 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 406927us
I20250905 19:59:44.506744 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:44.500741 28158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 1902) took 1733 ms. Trace:
I20250905 19:59:44.486486 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:44.512003 28001 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 1.
I20250905 19:59:44.510308 28158 rpcz_store.cc:276] 0905 19:59:42.767765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.469358 (+1701593us) service_pool.cc:225] Handling call
0905 19:59:44.500685 (+ 31327us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:44.513567 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:44.514408 28157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 1911) took 1087 ms. Trace:
I20250905 19:59:44.514736 28157 rpcz_store.cc:276] 0905 19:59:43.427224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.488330 (+1061106us) service_pool.cc:225] Handling call
0905 19:59:44.514318 (+ 25988us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:44.340569 29805 consensus_queue.cc:237] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:44.452582 28002 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1841) took 1835 ms (client timeout 1956 ms). Trace:
W20250905 19:59:44.517969 28002 rpcz_store.cc:269] 0905 19:59:42.617291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:43.245540 (+628249us) service_pool.cc:225] Handling call
0905 19:59:44.452286 (+1206746us) spinlock_profiling.cc:243] Waited 455 ms on lock 0x7b54000cbf54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:44.452498 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1045583104}
W20250905 19:59:44.485612 28159 rpcz_store.cc:269] 0905 19:59:42.511386 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:42.512714 (+ 1328us) service_pool.cc:225] Handling call
0905 19:59:44.428135 (+1915421us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b5400200654. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:44.433069 (+ 4934us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2365212544}
I20250905 19:59:44.485283 29599 raft_consensus.cc:2747] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299) with higher term. Message: Invalid argument: T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 3. Current term is 4.
I20250905 19:59:44.491008 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:59:44.466722 28076 rpcz_store.cc:269] 0905 19:59:42.640826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:42.647879 (+ 7053us) service_pool.cc:225] Handling call
0905 19:59:44.414359 (+1766480us) spinlock_profiling.cc:243] Waited 689 ms on lock 0x7b540021f554. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:44.414540 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1584147200}
I20250905 19:59:44.493460 28080 raft_consensus.cc:2466] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:44.493793 28085 raft_consensus.cc:2391] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 3: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:44.494563 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:44.490029 27973 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:54164 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1871)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 184)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1876)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1872)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 180)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1241)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1244)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1252)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1873)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1877)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1856)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1245)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1878)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1869)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 188)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1246)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1866)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 189)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 187)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1870)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 182)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1256)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1259)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 181)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1263)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 190)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1862)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1879)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1880)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1269)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1270)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1266)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1272)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1865)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1867)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1851)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1238)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1843)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1847)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1855)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1859)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1248)
I20250905 19:59:44.393678 28081 raft_consensus.cc:2466] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
I20250905 19:59:44.498665 28717 leader_election.cc:304] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 19:59:44.503631 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:44.509274 27974 leader_election.cc:304] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:44.506242 27975 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:54164 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1871)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 184)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1876)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1872)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 180)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1241)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1244)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1252)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1873)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1877)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1856)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1245)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1878)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1869)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 188)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1246)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1866)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 189)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 187)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1870)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 182)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1256)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1259)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 181)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1263)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 190)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1862)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1879)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1880)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1269)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1270)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1266)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1881)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1272)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1865)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1867)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1851)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1238)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1843)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1847)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1855)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1859)
I20250905 19:59:44.520783 28151 raft_consensus.cc:2391] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 2: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:44.516991 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:44.487861 29881 raft_consensus.cc:513] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 19:59:44.504756 29801 raft_consensus.cc:2802] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:44.504961 29813 raft_consensus.cc:491] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:44.721195 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:44.595412 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:44.541442 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:44.751821 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 242636us
I20250905 19:59:44.735380 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:59:44.752921 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 254359us
I20250905 19:59:44.761389 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:44.761538 28717 leader_election.cc:304] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:44.729430 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:44.762279 28078 raft_consensus.cc:2391] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 4: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:44.553529 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 19:59:44.733794 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:44.759270 28159 raft_consensus.cc:2366] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a for earlier term 3. Current term is 4.
I20250905 19:59:44.750953 28082 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:44.512729 29737 leader_election.cc:290] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:44.768618 28001 raft_consensus.cc:2391] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:44.517558 29906 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:44.756192 28150 raft_consensus.cc:2466] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 19:59:44.759213 28157 raft_consensus.cc:3058] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:44.762307 27975 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1859) took 1300 ms. Trace:
I20250905 19:59:44.612448 29854 raft_consensus.cc:491] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:44.750480 28151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39586 (request call id 177) took 1297 ms. Trace:
I20250905 19:59:44.811486 28151 rpcz_store.cc:276] 0905 19:59:43.453183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.487305 (+1034122us) service_pool.cc:225] Handling call
0905 19:59:44.750399 (+263094us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:44.512950 29809 raft_consensus.cc:3058] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
W20250905 19:59:44.634701 29943 consensus_meta.cc:287] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.883s user 0.012s sys 0.000s
I20250905 19:59:44.763531 28158 raft_consensus.cc:2391] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
W20250905 19:59:44.763396 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:44.628213 29903 consensus_meta.cc:287] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.881s user 0.008s sys 0.004s
W20250905 19:59:44.742046 29973 log.cc:927] Time spent T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.273s user 0.000s sys 0.001s
I20250905 19:59:44.751886 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:44.591547 29934 raft_consensus.cc:513] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:44.500552 29928 leader_election.cc:290] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:44.767740 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:44.827515 28079 raft_consensus.cc:3058] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:44.768676 28007 raft_consensus.cc:2391] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:44.701615 28752 raft_consensus.cc:2466] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:44.769315 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:44.767717 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:44.827613 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:44.829967 28078 raft_consensus.cc:2391] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 6: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:44.830005 28151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 1906) took 1937 ms. Trace:
I20250905 19:59:44.830717 28151 rpcz_store.cc:276] 0905 19:59:42.892696 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.816466 (+1923770us) service_pool.cc:225] Handling call
0905 19:59:44.829943 (+ 13477us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:44.755069 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:44.770181 27973 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1855) took 1637 ms. Trace:
W20250905 19:59:44.754667 29916 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.005s sys 0.005s
W20250905 19:59:44.758659 28746 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.004s sys 0.006s
I20250905 19:59:44.651178 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:44.717545 29645 leader_election.cc:290] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:44.771847 28001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 184) took 1174 ms. Trace:
I20250905 19:59:44.743165 29900 leader_election.cc:290] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:44.772961 27975 rpcz_store.cc:276] 0905 19:59:43.461756 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.762234 (+1300478us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250905 19:59:44.759878 29894 raft_consensus.cc:2802] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:44.757367 29763 raft_consensus.cc:2802] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:44.773679 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:44.774055 29976 log.cc:927] Time spent T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.189s user 0.001s sys 0.000s
W20250905 19:59:44.699941 28744 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40904 (request call id 489) took 2258 ms (client timeout 1877 ms). Trace:
I20250905 19:59:44.775633 29881 leader_election.cc:290] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:44.776743 28123 leader_election.cc:304] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:44.779209 28745 raft_consensus.cc:1238] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:44.762782 28002 raft_consensus.cc:2391] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 6: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:44.568889 29756 raft_consensus.cc:513] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:44.784497 28751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40904 (request call id 492) took 1919 ms. Trace:
I20250905 19:59:44.821910 29943 raft_consensus.cc:513] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:44.785594 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:44.789031 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 19:59:44.790464 28008 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 179) took 2402 ms (client timeout 1811 ms). Trace:
W20250905 19:59:44.844172 28006 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1247) took 1366 ms (client timeout 1658 ms). Trace:
W20250905 19:59:44.844323 28002 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1871) took 1212 ms (client timeout 1594 ms). Trace:
W20250905 19:59:44.844113 28008 rpcz_store.cc:269] 0905 19:59:42.388352 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:42.713166 (+324814us) service_pool.cc:225] Handling call
0905 19:59:44.790151 (+2076985us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b5400131fd4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:44.790355 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2806955776}
W20250905 19:59:44.844676 28006 rpcz_store.cc:269] 0905 19:59:43.478266 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.788650 (+1310384us) service_pool.cc:225] Handling call
0905 19:59:44.844111 (+ 55461us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:44.791524 28155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39586 (request call id 180) took 1287 ms. Trace:
I20250905 19:59:44.793187 28157 raft_consensus.cc:2466] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
W20250905 19:59:44.800400 28156 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 1888) took 2293 ms (client timeout 1969 ms). Trace:
I20250905 19:59:44.803939 29964 leader_election.cc:290] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:44.817745 28158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 1916) took 1193 ms. Trace:
W20250905 19:59:44.771332 29824 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.661s user 0.038s sys 0.078s
I20250905 19:59:44.819509 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:44.851123 28005 raft_consensus.cc:2391] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:44.822300 28080 raft_consensus.cc:3058] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:44.851604 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:44.851842 28079 raft_consensus.cc:2466] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
W20250905 19:59:44.852237 28006 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1242) took 1383 ms (client timeout 1776 ms). Trace:
W20250905 19:59:44.852479 28006 rpcz_store.cc:269] 0905 19:59:43.468411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.851303 (+1382892us) service_pool.cc:225] Handling call
0905 19:59:44.852193 (+ 890us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:44.745253 29917 consensus_meta.cc:287] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 1.475s user 0.007s sys 0.007s
I20250905 19:59:44.572516 29833 raft_consensus.cc:513] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:44.823899 28153 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 1891) took 2308 ms. Trace:
I20250905 19:59:44.581213 29826 raft_consensus.cc:513] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:44.831216 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:44.854161 28153 rpcz_store.cc:276] 0905 19:59:42.511116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:42.511230 (+ 114us) service_pool.cc:225] Handling call
0905 19:59:42.511533 (+ 303us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:44.195803 (+1684270us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:44.819106 (+623303us) spinlock_profiling.cc:243] Waited 1.64 s on lock 0x7b540009e2d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:44.819395 (+ 289us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3760614272}
I20250905 19:59:44.856351 28078 raft_consensus.cc:2391] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 3: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:44.832401 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:44.831918 27973 rpcz_store.cc:276] 0905 19:59:43.280776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.770116 (+1489340us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250905 19:59:44.857023 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:44.620750 29848 consensus_queue.cc:237] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:44.837016 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:44.836514 28001 rpcz_store.cc:276] 0905 19:59:43.597015 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.521651 (+924636us) service_pool.cc:225] Handling call
0905 19:59:44.771783 (+250132us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:44.617586 29871 consensus_queue.cc:237] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:44.838932 27975 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:54156 dropped due to backpressure: service queue is full with 50 items [suppressed 11 similar messages]
W20250905 19:59:44.838824 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 174667us
W20250905 19:59:44.833707 29916 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.650s user 0.006s sys 0.005s
I20250905 19:59:44.840819 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 19:59:44.833861 28746 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.593s user 0.005s sys 0.006s
I20250905 19:59:44.841630 28082 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1355) took 1411 ms. Trace:
I20250905 19:59:44.874621 28082 rpcz_store.cc:276] 0905 19:59:43.429924 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.766189 (+1336265us) service_pool.cc:225] Handling call
0905 19:59:44.841567 (+ 75378us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:44.843415 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 66762.4us
I20250905 19:59:44.845608 28157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 1915) took 1250 ms. Trace:
I20250905 19:59:44.845403 28155 rpcz_store.cc:276] 0905 19:59:43.504545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.142579 (+638034us) service_pool.cc:225] Handling call
0905 19:59:44.791278 (+648699us) spinlock_profiling.cc:243] Waited 100 ms on lock 0x7b54000e1ad4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:44.791450 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":230792576}
W20250905 19:59:44.845764 28156 rpcz_store.cc:269] 0905 19:59:42.510190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:42.510346 (+ 156us) service_pool.cc:225] Handling call
0905 19:59:44.800066 (+2289720us) spinlock_profiling.cc:243] Waited 1.86 s on lock 0x7b5400176e54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:44.800295 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4269453568}
I20250905 19:59:44.875666 28157 rpcz_store.cc:276] 0905 19:59:43.594868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.515371 (+920503us) service_pool.cc:225] Handling call
0905 19:59:44.845552 (+330181us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:44.844818 28002 rpcz_store.cc:269] 0905 19:59:43.632254 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.519232 (+886978us) service_pool.cc:225] Handling call
0905 19:59:44.844265 (+325033us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:44.823513 29903 raft_consensus.cc:513] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:44.846877 28158 rpcz_store.cc:276] 0905 19:59:43.624629 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.516258 (+891629us) service_pool.cc:225] Handling call
0905 19:59:44.817654 (+301396us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:44.847414 28004 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 183) took 1250 ms (client timeout 1552 ms). Trace:
I20250905 19:59:44.815580 29854 raft_consensus.cc:513] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:44.835855 28076 raft_consensus.cc:2466] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 1.
I20250905 19:59:44.882027 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:44.882193 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:44.882309 29756 leader_election.cc:290] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:44.882885 28156 raft_consensus.cc:2391] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
W20250905 19:59:44.883406 28051 leader_election.cc:336] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.520s (SENT)
I20250905 19:59:44.883906 28051 leader_election.cc:304] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:44.852829 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:44.881356 28123 leader_election.cc:304] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 19:59:44.882889 28155 raft_consensus.cc:2391] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 5: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
W20250905 19:59:44.854873 28005 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1863) took 1322 ms (client timeout 1590 ms). Trace:
I20250905 19:59:44.843305 28751 rpcz_store.cc:276] 0905 19:59:42.865253 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:43.089549 (+224296us) service_pool.cc:225] Handling call
0905 19:59:43.445404 (+355855us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:43.445797 (+ 393us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:43.445831 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:43.446338 (+ 507us) log.cc:844] Serialized 26 byte log entry
0905 19:59:43.762503 (+316165us) raft_consensus.cc:1643] Marking committed up to 4
0905 19:59:43.762558 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:43.762634 (+ 76us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:43.816175 (+ 53541us) raft_consensus.cc:1683] finished
0905 19:59:43.816207 (+ 32us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:44.784183 (+967976us) spinlock_profiling.cc:243] Waited 306 ms on lock 0x7b540069c8d0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:44.784415 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":172,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":102,"spinlock_wait_cycles":703677184,"thread_start_us":315692,"threads_started":1,"wal-append.queue_time_us":316218,"wal-append.run_cpu_time_us":2192,"wal-append.run_wall_time_us":1062859}
W20250905 19:59:44.885740 28005 rpcz_store.cc:269] 0905 19:59:43.532072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.819125 (+1287053us) service_pool.cc:225] Handling call
0905 19:59:44.854810 (+ 35685us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:44.886595 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:59:44.887158 28005 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1241) took 1419 ms (client timeout 1821 ms). Trace:
I20250905 19:59:44.887265 28080 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
W20250905 19:59:44.887392 28005 rpcz_store.cc:269] 0905 19:59:43.468067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.886330 (+1418263us) service_pool.cc:225] Handling call
0905 19:59:44.887115 (+ 785us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:44.888067 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:59:44.888536 28005 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1244) took 1419 ms (client timeout 1831 ms). Trace:
W20250905 19:59:44.888748 28005 rpcz_store.cc:269] 0905 19:59:43.477261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.887849 (+1410588us) service_pool.cc:225] Handling call
0905 19:59:44.888498 (+ 649us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:44.889319 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:44.840395 29763 raft_consensus.cc:491] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:44.839630 29894 raft_consensus.cc:491] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:44.765443 29813 raft_consensus.cc:513] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:44.857542 28081 raft_consensus.cc:2391] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 3: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:44.859440 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:59:44.898655 28008 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1876) took 1239 ms (client timeout 1574 ms). Trace:
W20250905 19:59:44.899204 28008 rpcz_store.cc:269] 0905 19:59:43.659393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.845251 (+1185858us) service_pool.cc:225] Handling call
0905 19:59:44.898591 (+ 53340us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:44.850736 29824 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.745s user 0.039s sys 0.079s
I20250905 19:59:44.857502 27973 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:54156 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 180)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1241)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1244)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1252)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1873)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1877)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1856)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1245)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1878)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1869)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 188)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1246)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1866)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 189)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 187)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1870)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 182)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1256)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1259)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 181)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1263)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 190)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1862)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1879)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1880)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1269)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1275)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1270)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1266)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1881)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1272)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1865)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1273)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1882)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1274)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1867)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1851)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1238)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1843)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1847)
I20250905 19:59:44.860981 28159 raft_consensus.cc:2366] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4.
I20250905 19:59:44.846909 29645 raft_consensus.cc:2747] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:44.904594 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 19:59:44.873811 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:44.876731 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:44.905411 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:44.905706 28157 raft_consensus.cc:2366] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 6. Current term is 7.
I20250905 19:59:44.905995 28153 raft_consensus.cc:2466] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
I20250905 19:59:44.877806 28050 leader_election.cc:304] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:44.905436 28082 raft_consensus.cc:3058] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 4
I20250905 19:59:44.860344 29853 consensus_queue.cc:1046] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.019s
I20250905 19:59:44.880723 28009 raft_consensus.cc:2391] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:44.853098 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:44.907357 28006 raft_consensus.cc:2366] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 6. Current term is 7.
I20250905 19:59:44.884219 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:44.892750 28005 raft_consensus.cc:2391] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:44.873001 28078 raft_consensus.cc:1238] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 3. Current term is 4. Ops: []
W20250905 19:59:44.877575 28004 rpcz_store.cc:269] 0905 19:59:43.596676 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.785254 (+1188578us) service_pool.cc:225] Handling call
0905 19:59:44.847363 (+ 62109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:44.908216 28078 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1339) took 2261 ms. Trace:
I20250905 19:59:44.908375 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:44.908569 28078 rpcz_store.cc:276] 0905 19:59:42.647260 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.860816 (+2213556us) service_pool.cc:225] Handling call
0905 19:59:44.872780 (+ 11964us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:44.907983 (+ 35203us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:44.908168 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:44.908990 28004 raft_consensus.cc:2466] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
I20250905 19:59:44.909435 28004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1877) took 1249 ms. Trace:
I20250905 19:59:44.909719 28004 rpcz_store.cc:276] 0905 19:59:43.659760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.908139 (+1248379us) service_pool.cc:225] Handling call
0905 19:59:44.909384 (+ 1245us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:44.910676 28051 leader_election.cc:336] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.545s (SENT)
I20250905 19:59:44.905731 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 19:59:44.912243 28006 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1872) took 1267 ms (client timeout 1611 ms). Trace:
W20250905 19:59:44.917388 28006 rpcz_store.cc:269] 0905 19:59:43.644470 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.852890 (+1208420us) service_pool.cc:225] Handling call
0905 19:59:44.912155 (+ 59265us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:44.917765 28002 raft_consensus.cc:2391] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 2: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
W20250905 19:59:44.918493 28002 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1240) took 1450 ms (client timeout 1819 ms). Trace:
I20250905 19:59:44.918661 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 19:59:44.919023 28002 rpcz_store.cc:269] 0905 19:59:43.467657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.877390 (+1409733us) service_pool.cc:225] Handling call
0905 19:59:44.918377 (+ 40987us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:44.855391 29917 raft_consensus.cc:513] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:44.923061 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:44.923390 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:44.924469 28004 raft_consensus.cc:2366] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4.
I20250905 19:59:44.913282 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:59:44.924878 28004 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1856) took 1590 ms (client timeout 2111 ms). Trace:
I20250905 19:59:44.925668 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:44.927541 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:44.928452 28156 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
I20250905 19:59:44.929081 28156 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 1894) took 2400 ms. Trace:
I20250905 19:59:44.929473 28156 rpcz_store.cc:276] 0905 19:59:42.528359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.927258 (+2398899us) service_pool.cc:225] Handling call
0905 19:59:44.929015 (+ 1757us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:44.914531 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:44.915431 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:44.931488 28151 raft_consensus.cc:2391] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 4: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:44.933902 28151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:33742 (request call id 638) took 2422 ms. Trace:
I20250905 19:59:44.916597 28079 raft_consensus.cc:1238] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:44.919874 28006 raft_consensus.cc:2391] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 4: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:44.934404 28151 rpcz_store.cc:276] 0905 19:59:42.511919 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.932280 (+2420361us) service_pool.cc:225] Handling call
0905 19:59:44.933819 (+ 1539us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:44.935315 28079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1341) took 2183 ms. Trace:
I20250905 19:59:44.938612 28079 rpcz_store.cc:276] 0905 19:59:42.752188 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.885104 (+2132916us) service_pool.cc:225] Handling call
0905 19:59:44.916338 (+ 31234us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:44.934944 (+ 18606us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:44.935238 (+ 294us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:44.925360 28004 rpcz_store.cc:269] 0905 19:59:43.334482 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.923036 (+1588554us) service_pool.cc:225] Handling call
0905 19:59:44.924823 (+ 1787us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:44.942009 28051 leader_election.cc:336] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.574s (SENT)
I20250905 19:59:44.941350 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:44.944756 28004 raft_consensus.cc:2391] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:44.945474 28004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1869) took 1330 ms. Trace:
I20250905 19:59:44.945991 28004 rpcz_store.cc:276] 0905 19:59:43.614977 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.941030 (+1326053us) service_pool.cc:225] Handling call
0905 19:59:44.945392 (+ 4362us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:44.935858 28006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1245) took 1458 ms. Trace:
I20250905 19:59:44.948249 28006 rpcz_store.cc:276] 0905 19:59:43.477620 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.918297 (+1440677us) service_pool.cc:225] Handling call
0905 19:59:44.935778 (+ 17481us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:44.949033 28005 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1252) took 1352 ms (client timeout 1711 ms). Trace:
I20250905 19:59:44.949452 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:59:44.949637 28005 rpcz_store.cc:269] 0905 19:59:43.596278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.889148 (+1292870us) service_pool.cc:225] Handling call
0905 19:59:44.948947 (+ 59799us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:44.950683 28006 raft_consensus.cc:2466] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 2.
I20250905 19:59:44.951287 28151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 1900) took 2238 ms. Trace:
I20250905 19:59:44.951663 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:44.947618 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:44.952802 28006 raft_consensus.cc:2391] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 6: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:44.954416 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:44.882345 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:44.959262 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:44.959883 28076 raft_consensus.cc:1238] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20250905 19:59:44.960270 28157 raft_consensus.cc:2466] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:44.960631 28076 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1316) took 2092 ms. Trace:
I20250905 19:59:44.960940 28157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 1918) took 1268 ms. Trace:
I20250905 19:59:44.961089 28076 rpcz_store.cc:276] 0905 19:59:42.868698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.959230 (+2090532us) service_pool.cc:225] Handling call
0905 19:59:44.959618 (+ 388us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:44.960337 (+ 719us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:44.960570 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:44.962700 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:44.897552 29763 raft_consensus.cc:3058] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:44.951906 28151 rpcz_store.cc:276] 0905 19:59:42.712855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.938110 (+2225255us) service_pool.cc:225] Handling call
0905 19:59:44.950732 (+ 12622us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:44.950970 (+ 238us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:44.950997 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:44.951042 (+ 45us) raft_consensus.cc:1643] Marking committed up to 7
0905 19:59:44.951076 (+ 34us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:44.951145 (+ 69us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:44.951236 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:44.955551 28051 leader_election.cc:336] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.594s (SENT)
I20250905 19:59:44.957466 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:44.968616 28159 raft_consensus.cc:2366] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4.
I20250905 19:59:44.961588 28157 rpcz_store.cc:276] 0905 19:59:43.692631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.920901 (+1228270us) service_pool.cc:225] Handling call
0905 19:59:44.960868 (+ 39967us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:44.951709 28156 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
from {username='slave'} at 127.0.0.1:36164
W20250905 19:59:44.842760 28744 rpcz_store.cc:269] 0905 19:59:42.441769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:42.447464 (+ 5695us) service_pool.cc:225] Handling call
0905 19:59:44.699598 (+2252134us) spinlock_profiling.cc:243] Waited 996 ms on lock 0x7b54004321d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:44.699820 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2290720512}
W20250905 19:59:44.964743 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 203274us
I20250905 19:59:44.890681 29862 raft_consensus.cc:2747] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:44.901201 29894 raft_consensus.cc:3058] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:44.951920 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:45.004015 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:45.004742 28158 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
I20250905 19:59:45.004707 28005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1246) took 1526 ms. Trace:
I20250905 19:59:45.005657 28005 rpcz_store.cc:276] 0905 19:59:43.477974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.950635 (+1472661us) service_pool.cc:225] Handling call
0905 19:59:45.004649 (+ 54014us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:44.963573 28008 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1873) took 1313 ms (client timeout 1709 ms). Trace:
I20250905 19:59:44.978700 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:44.952910 28004 raft_consensus.cc:2466] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
W20250905 19:59:45.007385 28051 leader_election.cc:336] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.611s (SENT)
I20250905 19:59:45.007009 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:45.000185 28156 raft_consensus.cc:604] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Received request to transfer leadership to TS 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:45.003584 28010 raft_consensus.cc:2366] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4.
I20250905 19:59:44.881098 29644 consensus_queue.cc:1046] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.079s
I20250905 19:59:44.901770 29928 raft_consensus.cc:2802] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:45.008076 28005 raft_consensus.cc:3058] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 6
I20250905 19:59:45.086647 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:45.125633 28717 leader_election.cc:400] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 election: Vote denied by peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705) with higher term. Message: Invalid argument: T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a for earlier term 3. Current term is 4.
I20250905 19:59:45.136763 28752 raft_consensus.cc:2366] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6.
I20250905 19:59:45.137171 28717 leader_election.cc:403] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20250905 19:59:45.006146 28006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1866) took 1440 ms. Trace:
I20250905 19:59:45.139425 28006 rpcz_store.cc:276] 0905 19:59:43.565688 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.954015 (+1388327us) service_pool.cc:225] Handling call
0905 19:59:45.006085 (+ 52070us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:45.121369 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 243645us
I20250905 19:59:45.141016 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:45.141960 28006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1870) took 1515 ms. Trace:
W20250905 19:59:45.084955 28747 consensus_meta.cc:287] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.941s user 0.012s sys 0.002s
I20250905 19:59:45.142524 28006 rpcz_store.cc:276] 0905 19:59:43.626223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.140553 (+1514330us) service_pool.cc:225] Handling call
0905 19:59:45.141902 (+ 1349us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:45.069641 28051 leader_election.cc:336] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.709s (SENT)
I20250905 19:59:45.063110 28157 raft_consensus.cc:1238] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 LEADER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
W20250905 19:59:45.090493 28718 leader_election.cc:336] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.552s (SENT)
I20250905 19:59:45.152297 28158 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 6.
I20250905 19:59:45.157928 28158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 1910) took 1792 ms. Trace:
I20250905 19:59:45.159524 28158 rpcz_store.cc:276] 0905 19:59:43.366009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.922676 (+1556667us) service_pool.cc:225] Handling call
0905 19:59:45.157862 (+235186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:45.161680 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:45.161985 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:45.008638 28156 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:36164 (request call id 12) took 2153 ms. Trace:
I20250905 19:59:45.008205 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:59:45.120178 29981 log.cc:927] Time spent T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.206s user 0.000s sys 0.001s
W20250905 19:59:45.151346 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 81784.7us
I20250905 19:59:44.961651 29936 raft_consensus.cc:513] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:45.152818 28718 leader_election.cc:304] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:45.073251 28151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 1903) took 2292 ms. Trace:
I20250905 19:59:44.966348 28077 raft_consensus.cc:2391] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 4: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
W20250905 19:59:44.963105 29883 consensus_meta.cc:287] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 1.157s user 0.014s sys 0.000s
I20250905 19:59:45.153319 27973 leader_election.cc:304] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:45.152230 28157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 1913) took 1658 ms. Trace:
I20250905 19:59:45.156064 28082 raft_consensus.cc:2466] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
W20250905 19:59:45.146502 28125 consensus_peers.cc:489] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 -> Peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Couldn't send request to peer cd764bfeebac4cc9ad851cd4a6a7d3b0. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:54164 dropped due to backpressure: service queue is full with 50 items. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:45.006990 28002 raft_consensus.cc:2466] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
W20250905 19:59:45.191926 28002 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1878) took 1531 ms (client timeout 1812 ms). Trace:
W20250905 19:59:45.006773 28008 rpcz_store.cc:269] 0905 19:59:43.650866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.900093 (+1249227us) service_pool.cc:225] Handling call
0905 19:59:44.963511 (+ 63418us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:45.162704 28006 raft_consensus.cc:2391] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:45.163342 28081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1309) took 2323 ms. Trace:
I20250905 19:59:45.127359 29903 leader_election.cc:290] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:45.109606 29826 leader_election.cc:290] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:45.164489 28156 rpcz_store.cc:276] 0905 19:59:42.855160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.930381 (+2075221us) service_pool.cc:225] Handling call
0905 19:59:45.008546 (+ 78165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24704}
W20250905 19:59:45.170152 28051 leader_election.cc:336] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.782s (SENT)
I20250905 19:59:45.144520 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:45.143638 29801 raft_consensus.cc:695] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:45.142730 28747 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 19:59:45.154933 29862 raft_consensus.cc:2802] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:45.154527 29884 raft_consensus.cc:491] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:45.178992 28151 rpcz_store.cc:276] 0905 19:59:42.781124 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.967671 (+2186547us) service_pool.cc:225] Handling call
0905 19:59:45.072621 (+104950us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:45.072813 (+ 192us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:45.072841 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:45.072901 (+ 60us) raft_consensus.cc:1643] Marking committed up to 4
0905 19:59:45.072948 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:45.073028 (+ 80us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:45.073172 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:45.185557 28157 rpcz_store.cc:276] 0905 19:59:43.493782 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.969397 (+1475615us) service_pool.cc:225] Handling call
0905 19:59:45.062760 (+ 93363us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:45.151938 (+ 89178us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:45.152172 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:45.192337 28002 rpcz_store.cc:269] 0905 19:59:43.660043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.919945 (+1259902us) service_pool.cc:225] Handling call
0905 19:59:45.191863 (+271918us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:45.193271 28081 rpcz_store.cc:276] 0905 19:59:42.840055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.905827 (+2065772us) service_pool.cc:225] Handling call
0905 19:59:44.906109 (+ 282us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:45.154893 (+248784us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:45.154925 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:45.162997 (+ 8072us) raft_consensus.cc:1643] Marking committed up to 2
0905 19:59:45.163059 (+ 62us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:45.163148 (+ 89us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:45.163289 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":247248,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":102,"thread_start_us":239088,"threads_started":1}
W20250905 19:59:45.194166 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 47738.3us
I20250905 19:59:45.195132 27973 leader_election.cc:304] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
W20250905 19:59:45.198663 28006 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1846) took 2523 ms (client timeout 3177 ms). Trace:
I20250905 19:59:45.162343 29928 raft_consensus.cc:695] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:45.166204 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
W20250905 19:59:45.208935 28051 leader_election.cc:336] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.812s (SENT)
I20250905 19:59:45.214412 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:45.168654 29934 leader_election.cc:290] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:45.217432 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:45.219673 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250905 19:59:45.198009 29943 leader_election.cc:290] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:45.179463 29987 raft_consensus.cc:2802] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:45.222064 29853 raft_consensus.cc:491] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:45.260376 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 51478.6us
I20250905 19:59:45.207922 29906 raft_consensus.cc:2802] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250905 19:59:45.289851 28125 consensus_peers.cc:489] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 -> Peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Couldn't send request to peer cd764bfeebac4cc9ad851cd4a6a7d3b0. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:54164 dropped due to backpressure: service queue is full with 50 items. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:59:45.306403 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:45.298262 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:45.306754 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 19:59:45.312536 28051 leader_election.cc:336] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.827s (SENT)
I20250905 19:59:45.222470 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:45.290802 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 19:59:45.236624 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:45.222730 28080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1342) took 2465 ms. Trace:
I20250905 19:59:45.260967 29833 leader_election.cc:290] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 19:59:45.261806 28006 rpcz_store.cc:269] 0905 19:59:42.675274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.160130 (+2484856us) service_pool.cc:225] Handling call
0905 19:59:45.198567 (+ 38437us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:45.371798 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 281371us
W20250905 19:59:45.372189 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:45.372457 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 19:59:45.263293 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 68234.3us
I20250905 19:59:45.371447 28080 rpcz_store.cc:276] 0905 19:59:42.757007 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.887784 (+2130777us) service_pool.cc:225] Handling call
0905 19:59:44.888004 (+ 220us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:44.888144 (+ 140us) raft_consensus.cc:1532] Early marking committed up to index 2
0905 19:59:45.114015 (+225871us) log.cc:844] Serialized 14 byte log entry
0905 19:59:45.221953 (+107938us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:45.222238 (+ 285us) raft_consensus.cc:1643] Marking committed up to 2
0905 19:59:45.222379 (+ 141us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:45.222480 (+ 101us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:45.222657 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":317416,"threads_started":2}
I20250905 19:59:45.375350 28002 raft_consensus.cc:2391] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:45.374308 28006 raft_consensus.cc:2391] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 7: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:45.373634 28156 raft_consensus.cc:2391] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 4: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:45.373668 28051 leader_election.cc:304] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:45.288933 29903 raft_consensus.cc:2747] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250905 19:59:45.377732 28002 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1256) took 1695 ms (client timeout 2239 ms). Trace:
I20250905 19:59:45.378651 28006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1259) took 1342 ms. Trace:
I20250905 19:59:45.379745 28006 rpcz_store.cc:276] 0905 19:59:44.035769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.372079 (+1336310us) service_pool.cc:225] Handling call
0905 19:59:45.378599 (+ 6520us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:45.208701 29864 raft_consensus.cc:491] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:45.373654 28718 leader_election.cc:336] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.607s (SENT)
W20250905 19:59:45.378537 28002 rpcz_store.cc:269] 0905 19:59:43.682000 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.260891 (+1578891us) service_pool.cc:225] Handling call
0905 19:59:45.377660 (+116769us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:45.380965 28718 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:45.379331 28156 raft_consensus.cc:1273] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:59:45.381399 28081 raft_consensus.cc:2391] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 3: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
W20250905 19:59:45.373898 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 84141.4us
I20250905 19:59:45.382061 28157 raft_consensus.cc:2391] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
W20250905 19:59:45.350320 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 224910us
I20250905 19:59:45.384020 28156 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 1917) took 1729 ms. Trace:
I20250905 19:59:45.384675 28156 rpcz_store.cc:276] 0905 19:59:43.655009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.378714 (+1723705us) service_pool.cc:225] Handling call
0905 19:59:45.379029 (+ 315us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:45.381717 (+ 2688us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:45.383942 (+ 2225us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:45.382978 28008 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 182) took 1870 ms (client timeout 2342 ms). Trace:
I20250905 19:59:45.382941 28125 leader_election.cc:304] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
W20250905 19:59:45.386255 28008 rpcz_store.cc:269] 0905 19:59:43.512506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.195081 (+1682575us) service_pool.cc:225] Handling call
0905 19:59:45.382904 (+187823us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:45.293004 29982 log.cc:927] Time spent T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.439s user 0.001s sys 0.000s
I20250905 19:59:45.263831 29854 leader_election.cc:290] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:45.313745 29862 raft_consensus.cc:695] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:45.408812 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 19:59:45.434551 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 60923.1us
I20250905 19:59:45.435249 28081 raft_consensus.cc:2391] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 7: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:45.436384 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:45.393020 28156 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 1921) took 1241 ms. Trace:
I20250905 19:59:45.436969 28081 raft_consensus.cc:2466] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:59:45.392076 28125 leader_election.cc:304] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 19:59:45.391247 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 78751.4us
W20250905 19:59:45.439342 28051 leader_election.cc:336] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.836s (SENT)
W20250905 19:59:45.440172 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 48226.9us
I20250905 19:59:45.438871 28156 rpcz_store.cc:276] 0905 19:59:44.152127 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.386030 (+1233903us) service_pool.cc:225] Handling call
0905 19:59:45.392334 (+ 6304us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:45.392575 (+ 241us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:45.392615 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:45.392677 (+ 62us) raft_consensus.cc:1643] Marking committed up to 4
0905 19:59:45.392725 (+ 48us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:45.392800 (+ 75us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:45.392950 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:45.440333 28051 leader_election.cc:336] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.956s (SENT)
W20250905 19:59:45.443552 28051 leader_election.cc:336] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.969s (SENT)
I20250905 19:59:45.443650 28158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39586 (request call id 183) took 1855 ms. Trace:
I20250905 19:59:45.409356 28050 leader_election.cc:400] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705) with higher term. Message: Invalid argument: T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 6. Current term is 7.
I20250905 19:59:45.445338 28158 rpcz_store.cc:276] 0905 19:59:43.588627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.377079 (+1788452us) service_pool.cc:225] Handling call
0905 19:59:45.443168 (+ 66089us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:45.443332 (+ 164us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:45.443357 (+ 25us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:45.443400 (+ 43us) raft_consensus.cc:1643] Marking committed up to 6
0905 19:59:45.443426 (+ 26us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:45.443476 (+ 50us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:45.443598 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:45.304677 29983 log.cc:927] Time spent T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.277s user 0.000s sys 0.001s
I20250905 19:59:45.444690 28125 leader_election.cc:304] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 19:59:45.479465 28152 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39586 (request call id 176) took 2026 ms (client timeout 1713 ms). Trace:
W20250905 19:59:45.480098 28152 rpcz_store.cc:269] 0905 19:59:43.452845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:43.853326 (+400481us) service_pool.cc:225] Handling call
0905 19:59:45.479154 (+1625828us) spinlock_profiling.cc:243] Waited 364 ms on lock 0x7b54000d20d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:45.479372 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":836661632}
I20250905 19:59:45.437387 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:45.291549 28747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40926 (request call id 391) took 1195 ms. Trace:
W20250905 19:59:45.448071 28051 leader_election.cc:336] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 2.013s (SENT)
I20250905 19:59:45.448643 28718 leader_election.cc:304] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:45.442185 28156 raft_consensus.cc:1273] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:59:45.313840 29906 raft_consensus.cc:491] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:45.442724 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:45.442834 29824 raft_consensus.cc:491] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:45.288100 29884 raft_consensus.cc:513] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:45.458880 28158 raft_consensus.cc:1273] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250905 19:59:45.451100 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:45.497506 28081 raft_consensus.cc:3058] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:45.451972 28746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40918 (request call id 438) took 2043 ms. Trace:
I20250905 19:59:45.394856 27974 leader_election.cc:400] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299) with higher term. Message: Invalid argument: T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6.
I20250905 19:59:45.505486 27974 leader_election.cc:403] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
W20250905 19:59:45.506029 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 111382us
I20250905 19:59:45.492980 28747 rpcz_store.cc:276] 0905 19:59:44.092782 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.104586 (+ 11804us) service_pool.cc:225] Handling call
0905 19:59:45.287872 (+1183286us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:45.452073 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:45.507652 28008 raft_consensus.cc:2466] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 2.
I20250905 19:59:45.508132 28008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 190) took 1473 ms. Trace:
I20250905 19:59:45.508258 28750 raft_consensus.cc:1238] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:45.508435 28008 rpcz_store.cc:276] 0905 19:59:44.034914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.392484 (+1357570us) service_pool.cc:225] Handling call
0905 19:59:45.508068 (+115584us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:45.452198 29999 raft_consensus.cc:2747] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:45.509258 28750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40904 (request call id 496) took 1432 ms. Trace:
I20250905 19:59:45.460533 29903 consensus_queue.cc:1046] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20250905 19:59:45.509685 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:45.495676 29906 raft_consensus.cc:3058] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:45.447615 29943 consensus_queue.cc:1046] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20250905 19:59:45.222335 29883 raft_consensus.cc:3053] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:45.263567 28748 raft_consensus.cc:1238] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:45.297681 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:45.410329 29864 raft_consensus.cc:513] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:45.469127 29928 consensus_queue.cc:237] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:45.361295 29989 raft_consensus.cc:2747] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705) with higher term. Message: Invalid argument: T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a for earlier term 3. Current term is 4.
I20250905 19:59:45.310928 29987 raft_consensus.cc:491] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:45.360602 29917 leader_election.cc:290] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:45.407068 29619 raft_consensus.cc:2747] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:45.440392 29996 raft_consensus.cc:2747] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:45.567286 29996 raft_consensus.cc:2802] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:45.567508 29840 leader_election.cc:290] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:45.567773 29996 raft_consensus.cc:491] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:45.568302 29996 raft_consensus.cc:3058] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:45.481628 28125 leader_election.cc:304] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
W20250905 19:59:45.570083 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 88569.1us
I20250905 19:59:45.551162 29936 leader_election.cc:290] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:45.572336 28125 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:45.551690 29992 raft_consensus.cc:3058] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:45.503248 29824 raft_consensus.cc:513] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:45.491474 28006 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 181) took 2064 ms (client timeout 2699 ms). Trace:
W20250905 19:59:45.574590 28006 rpcz_store.cc:269] 0905 19:59:43.427180 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.409013 (+1981833us) service_pool.cc:225] Handling call
0905 19:59:45.491400 (+ 82387us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:45.575505 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:45.576201 28006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1257) took 1541 ms. Trace:
W20250905 19:59:45.576233 29931 consensus_meta.cc:287] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 1.056s user 0.004s sys 0.009s
I20250905 19:59:45.576570 28006 rpcz_store.cc:276] 0905 19:59:44.035218 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.575249 (+1540031us) service_pool.cc:225] Handling call
0905 19:59:45.576151 (+ 902us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:45.577608 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:45.294942 29813 leader_election.cc:290] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:45.448483 29964 raft_consensus.cc:2747] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:45.492779 28002 raft_consensus.cc:2391] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:45.578533 29854 raft_consensus.cc:2747] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 19:59:45.578728 28002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1263) took 1387 ms. Trace:
W20250905 19:59:45.491896 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 43866.2us
I20250905 19:59:45.579099 28002 rpcz_store.cc:276] 0905 19:59:44.191174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.381388 (+1190214us) service_pool.cc:225] Handling call
0905 19:59:45.578666 (+197278us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:45.579998 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:45.374884 28744 raft_consensus.cc:2391] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 4: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 19:59:45.580523 28002 raft_consensus.cc:2374] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3. Re-sending same reply.
I20250905 19:59:45.580996 28002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1880) took 1155 ms. Trace:
I20250905 19:59:45.348735 29853 raft_consensus.cc:513] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:45.451578 28050 leader_election.cc:403] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
W20250905 19:59:45.493435 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 44899.7us
I20250905 19:59:45.504473 28152 raft_consensus.cc:1273] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250905 19:59:45.478992 29901 consensus_meta.cc:287] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.265s user 0.007s sys 0.006s
W20250905 19:59:45.582371 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 173232us
I20250905 19:59:45.582736 29901 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:45.583237 28050 leader_election.cc:400] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Vote denied by peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705) with higher term. Message: Invalid argument: T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4.
I20250905 19:59:45.583518 28050 leader_election.cc:403] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
W20250905 19:59:45.501331 28003 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1844) took 2872 ms (client timeout 2013 ms). Trace:
W20250905 19:59:45.583964 28003 rpcz_store.cc:269] 0905 19:59:42.629352 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:43.373601 (+744249us) service_pool.cc:225] Handling call
0905 19:59:45.501115 (+2127514us) spinlock_profiling.cc:243] Waited 838 ms on lock 0x7b54003417d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:45.501261 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1927176576}
I20250905 19:59:45.584432 28006 raft_consensus.cc:2466] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
W20250905 19:59:45.508909 29809 consensus_meta.cc:287] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.692s user 0.009s sys 0.000s
I20250905 19:59:45.451977 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 19:59:45.373744 29511 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.225s user 0.016s sys 0.014s
I20250905 19:59:45.509737 28750 rpcz_store.cc:276] 0905 19:59:44.077075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.077527 (+ 452us) service_pool.cc:225] Handling call
0905 19:59:45.502288 (+1424761us) spinlock_profiling.cc:243] Waited 21 ms on lock 0x7b5800304598. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b91f60 00007fe4a2b7d1a9 00007fe4a2b7cc52 00007fe49c202b44
0905 19:59:45.502529 (+ 241us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:45.508943 (+ 6414us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:45.509188 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":48411392}
W20250905 19:59:45.548682 28084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1337) took 2889 ms (client timeout 2872 ms). Trace:
I20250905 19:59:45.585361 28154 raft_consensus.cc:2391] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 5: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
W20250905 19:59:45.597980 29511 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.449s user 0.021s sys 0.016s
I20250905 19:59:45.558640 29987 raft_consensus.cc:3058] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:45.591902 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:45.585337 29997 raft_consensus.cc:3058] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:45.599280 28750 raft_consensus.cc:1238] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:45.567867 29884 leader_election.cc:290] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:45.491011 28156 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 1925) took 1134 ms. Trace:
I20250905 19:59:45.570711 29934 raft_consensus.cc:2747] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250905 19:59:45.600768 29824 leader_election.cc:290] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:45.581272 28002 rpcz_store.cc:276] 0905 19:59:44.429108 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.579759 (+1150651us) service_pool.cc:225] Handling call
0905 19:59:45.580947 (+ 1188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:45.589423 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:45.602128 30004 consensus_queue.cc:1046] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250905 19:59:45.600481 28156 rpcz_store.cc:276] 0905 19:59:44.356530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.441452 (+1084922us) service_pool.cc:225] Handling call
0905 19:59:45.441787 (+ 335us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:45.490708 (+ 48921us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:45.490949 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:45.555305 29862 consensus_queue.cc:237] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:45.603348 28084 rpcz_store.cc:269] 0905 19:59:42.646628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:42.786159 (+139531us) service_pool.cc:225] Handling call
0905 19:59:43.348056 (+561897us) spinlock_profiling.cc:243] Waited 3.73 ms on lock 0x7b5800204f98. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b7db60 00007fe4a2b7d444 00007fe49c2050c4 00007fe49c205052
0905 19:59:45.535718 (+2187662us) spinlock_profiling.cc:243] Waited 506 ms on lock 0x7b540023ad54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:45.535905 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1171697920}
I20250905 19:59:45.604667 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:45.474915 29801 consensus_queue.cc:237] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:45.587195 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:45.606606 29809 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:45.588303 29854 raft_consensus.cc:991] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05: : Instructing follower 7e9c9f591f05464f9b097bf003d14f4a to start an election
I20250905 19:59:45.608096 28155 raft_consensus.cc:2391] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 5: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:45.608525 29854 raft_consensus.cc:1079] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Signalling peer 7e9c9f591f05464f9b097bf003d14f4a to start an election
I20250905 19:59:45.555006 29943 raft_consensus.cc:491] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:45.576879 29931 raft_consensus.cc:513] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:45.609386 29943 raft_consensus.cc:513] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:45.610270 29809 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:45.585665 29999 raft_consensus.cc:2747] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705) with higher term. Message: Invalid argument: T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 6. Current term is 7.
I20250905 19:59:45.602128 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:45.611418 29943 leader_election.cc:290] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:45.611891 28006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1879) took 1190 ms. Trace:
I20250905 19:59:45.530926 29883 raft_consensus.cc:738] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:45.612432 28083 raft_consensus.cc:2391] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 3: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:45.612457 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:45.612795 29883 consensus_queue.cc:260] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:45.613337 28745 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42"
dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
from {username='slave'} at 127.0.0.1:40926
I20250905 19:59:45.613668 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:45.614055 29883 raft_consensus.cc:3058] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:45.612282 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:45.614864 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:45.611491 29931 leader_election.cc:290] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:45.537330 28081 raft_consensus.cc:2466] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 19:59:45.615563 28006 rpcz_store.cc:276] 0905 19:59:44.421531 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.577311 (+1155780us) service_pool.cc:225] Handling call
0905 19:59:45.611830 (+ 34519us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:45.558395 28158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:33742 (request call id 639) took 1876 ms. Trace:
I20250905 19:59:45.616134 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:45.611624 28002 raft_consensus.cc:3058] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:45.605845 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:45.616930 28081 raft_consensus.cc:2391] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 3: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:45.599395 28003 raft_consensus.cc:2391] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:45.605813 28005 raft_consensus.cc:2466] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6.
W20250905 19:59:45.617606 28005 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 189) took 1591 ms (client timeout 1686 ms). Trace:
W20250905 19:59:45.618003 28005 rpcz_store.cc:269] 0905 19:59:44.026517 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.006680 (+980163us) service_pool.cc:225] Handling call
0905 19:59:45.617532 (+610852us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:45.618698 28078 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1310) took 2778 ms. Trace:
I20250905 19:59:45.619161 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:45.620018 28005 raft_consensus.cc:2466] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:45.620576 28005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1270) took 1241 ms. Trace:
I20250905 19:59:45.511121 29848 raft_consensus.cc:2747] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 19:59:45.584689 28050 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:45.621023 28005 rpcz_store.cc:276] 0905 19:59:44.379583 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.618838 (+1239255us) service_pool.cc:225] Handling call
0905 19:59:45.620493 (+ 1655us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:45.574064 28125 leader_election.cc:304] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:45.626986 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 19:59:45.627228 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 53229.1us
I20250905 19:59:45.627940 28005 raft_consensus.cc:3058] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:45.628778 28717 leader_election.cc:304] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
W20250905 19:59:45.635596 28003 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1243) took 2166 ms (client timeout 2814 ms). Trace:
I20250905 19:59:45.616379 28158 rpcz_store.cc:276] 0905 19:59:43.676669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.502933 (+1826264us) service_pool.cc:225] Handling call
0905 19:59:45.552898 (+ 49965us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:45.636181 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:45.636144 28003 rpcz_store.cc:269] 0905 19:59:43.468750 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.584856 (+2116106us) service_pool.cc:225] Handling call
0905 19:59:45.635523 (+ 50667us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:45.637194 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 245343us
I20250905 19:59:45.638103 28081 raft_consensus.cc:2408] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20250905 19:59:45.638801 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 54177.5us
I20250905 19:59:45.639788 28050 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:45.619179 28078 rpcz_store.cc:276] 0905 19:59:42.840368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.909325 (+2068957us) service_pool.cc:225] Handling call
0905 19:59:44.909621 (+ 296us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:45.190659 (+281038us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:45.190697 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:45.260138 (+ 69441us) log.cc:844] Serialized 26 byte log entry
0905 19:59:45.449737 (+189599us) raft_consensus.cc:1643] Marking committed up to 1
0905 19:59:45.449778 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:45.449845 (+ 67us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:45.618400 (+168555us) raft_consensus.cc:1683] finished
0905 19:59:45.618450 (+ 50us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:45.618628 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4344,"raft.queue_time_us":297294,"raft.run_cpu_time_us":230,"raft.run_wall_time_us":72051,"thread_start_us":344097,"threads_started":2,"wal-append.queue_time_us":297486}
I20250905 19:59:45.654531 28718 leader_election.cc:304] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:45.658202 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:45.660461 28005 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 19:59:45.642318 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:45.661492 28005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1266) took 1450 ms. Trace:
I20250905 19:59:45.661839 28081 raft_consensus.cc:1238] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:45.668054 28081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1357) took 2202 ms. Trace:
I20250905 19:59:45.668442 28717 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:45.667452 28005 rpcz_store.cc:276] 0905 19:59:44.211163 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.626586 (+1415423us) service_pool.cc:225] Handling call
0905 19:59:45.661433 (+ 34847us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:45.584762 29901 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:45.671092 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:45.668385 28081 rpcz_store.cc:276] 0905 19:59:43.466065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.661337 (+2195272us) service_pool.cc:225] Handling call
0905 19:59:45.661651 (+ 314us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:45.667782 (+ 6131us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:45.667996 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:45.674770 28050 leader_election.cc:400] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705) with higher term. Message: Invalid argument: T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4.
I20250905 19:59:45.674788 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:45.675282 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:45.675570 28050 leader_election.cc:403] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20250905 19:59:45.675860 28003 raft_consensus.cc:2391] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:45.676007 28005 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 19:59:45.676381 28003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1881) took 1179 ms. Trace:
I20250905 19:59:45.676504 28005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1272) took 1192 ms. Trace:
I20250905 19:59:45.676855 28005 rpcz_store.cc:276] 0905 19:59:44.484604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.670295 (+1185691us) service_pool.cc:225] Handling call
0905 19:59:45.676453 (+ 6158us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:45.676780 28003 rpcz_store.cc:276] 0905 19:59:44.496877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.637263 (+1140386us) service_pool.cc:225] Handling call
0905 19:59:45.676315 (+ 39052us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:45.678580 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:45.679220 29997 raft_consensus.cc:2747] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705) with higher term. Message: Invalid argument: T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4.
I20250905 19:59:45.679538 28005 raft_consensus.cc:2391] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 4: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:45.680027 28084 raft_consensus.cc:2391] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 5: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:45.680965 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:45.660660 29809 raft_consensus.cc:2802] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:45.682658 29809 raft_consensus.cc:491] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:45.682833 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:45.683643 29997 raft_consensus.cc:2747] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705) with higher term. Message: Invalid argument: T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4.
I20250905 19:59:45.683928 28003 raft_consensus.cc:2391] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 4: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:45.684610 28003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1865) took 2137 ms. Trace:
I20250905 19:59:45.683147 29809 raft_consensus.cc:3058] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:45.685261 28050 leader_election.cc:304] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:45.682760 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:45.686317 29997 raft_consensus.cc:2802] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:45.686797 29997 raft_consensus.cc:491] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:45.687144 29997 raft_consensus.cc:3058] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:45.687949 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:45.689150 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:45.689870 28005 raft_consensus.cc:2391] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 6: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:45.690259 28005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1867) took 2089 ms. Trace:
I20250905 19:59:45.661396 30002 raft_consensus.cc:2802] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:45.690936 30002 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:45.691350 30002 raft_consensus.cc:3058] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:45.685014 28003 rpcz_store.cc:276] 0905 19:59:43.546849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.677672 (+2130823us) service_pool.cc:225] Handling call
0905 19:59:45.684496 (+ 6824us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:45.635591 29924 raft_consensus.cc:2747] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299) with higher term. Message: Invalid argument: T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6.
I20250905 19:59:45.692116 28050 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:45.585590 29853 leader_election.cc:290] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:45.666107 29931 raft_consensus.cc:2747] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:45.697075 29999 raft_consensus.cc:2802] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election won for term 6
I20250905 19:59:45.697611 29999 raft_consensus.cc:695] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:45.699091 29999 consensus_queue.cc:237] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:45.701187 30005 raft_consensus.cc:2802] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:45.701747 30005 raft_consensus.cc:491] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:45.702131 30005 raft_consensus.cc:3058] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:45.702752 28079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1311) took 2862 ms. Trace:
I20250905 19:59:45.703178 28079 rpcz_store.cc:276] 0905 19:59:42.840721 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.939640 (+2098919us) service_pool.cc:225] Handling call
0905 19:59:44.940137 (+ 497us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:45.166625 (+226488us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:45.166656 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:45.171293 (+ 4637us) log.cc:844] Serialized 26 byte log entry
0905 19:59:45.393901 (+222608us) raft_consensus.cc:1643] Marking committed up to 1
0905 19:59:45.393953 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:45.394043 (+ 90us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:45.702523 (+308480us) raft_consensus.cc:1683] finished
0905 19:59:45.702550 (+ 27us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:45.702695 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27937,"raft.queue_time_us":279614,"spinlock_wait_cycles":150861568,"thread_start_us":304221,"threads_started":2,"wal-append.queue_time_us":447178}
I20250905 19:59:45.705307 28079 raft_consensus.cc:1273] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250905 19:59:45.705505 29931 consensus_queue.cc:1035] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:45.705928 28079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 176) took 1752 ms. Trace:
I20250905 19:59:45.706291 28079 rpcz_store.cc:276] 0905 19:59:43.953380 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.704633 (+1751253us) service_pool.cc:225] Handling call
0905 19:59:45.704986 (+ 353us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:45.705671 (+ 685us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:45.705875 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:45.707151 29939 leader_election.cc:290] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:45.715142 28715 leader_election.cc:304] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:45.717083 28715 leader_election.cc:304] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:45.673979 29989 raft_consensus.cc:2747] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 19:59:45.686215 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:45.740672 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:45.730829 28006 raft_consensus.cc:2391] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 6: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
W20250905 19:59:45.741264 28051 leader_election.cc:336] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.590s (SENT)
I20250905 19:59:45.690649 28005 rpcz_store.cc:276] 0905 19:59:43.601434 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.688899 (+2087465us) service_pool.cc:225] Handling call
0905 19:59:45.690203 (+ 1304us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:45.744414 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:45.744922 28003 raft_consensus.cc:2466] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:45.758083 28156 raft_consensus.cc:3058] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:45.656951 29943 consensus_queue.cc:1046] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
I20250905 19:59:45.745355 28051 leader_election.cc:304] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 19:59:45.758234 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 89857.5us
I20250905 19:59:45.751713 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:59:45.760461 28051 leader_election.cc:336] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.603s (SENT)
I20250905 19:59:45.760728 28005 raft_consensus.cc:1811] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Handling vote request from an unknown peer 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:45.802084 28005 raft_consensus.cc:2408] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
W20250905 19:59:45.728960 29941 consensus_meta.cc:287] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 1.307s user 0.014s sys 0.000s
I20250905 19:59:45.758291 28003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1261) took 1664 ms. Trace:
I20250905 19:59:45.745263 28752 raft_consensus.cc:1238] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 LEADER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:45.747987 28125 leader_election.cc:304] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:45.757001 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:45.760927 28051 leader_election.cc:304] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:45.811115 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:45.744127 27973 leader_election.cc:304] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:45.803676 28084 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 169) took 1989 ms. Trace:
I20250905 19:59:45.812007 28003 rpcz_store.cc:276] 0905 19:59:44.094015 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.692498 (+1598483us) service_pool.cc:225] Handling call
0905 19:59:45.758228 (+ 65730us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:45.821606 28084 rpcz_store.cc:276] 0905 19:59:43.814101 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.680687 (+1866586us) service_pool.cc:225] Handling call
0905 19:59:45.681044 (+ 357us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:45.681260 (+ 216us) raft_consensus.cc:1532] Early marking committed up to index 6
0905 19:59:45.681618 (+ 358us) log.cc:844] Serialized 14 byte log entry
0905 19:59:45.803326 (+121708us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:45.803383 (+ 57us) raft_consensus.cc:1643] Marking committed up to 6
0905 19:59:45.803425 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:45.803489 (+ 64us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:45.803618 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21453,"prepare.queue_time_us":195,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"thread_start_us":99804,"threads_started":1}
I20250905 19:59:45.823678 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:45.824528 28084 raft_consensus.cc:2391] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 3: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:45.825511 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:45.827167 29989 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:45.756047 29763 consensus_meta.cc:287] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.722s user 0.011s sys 0.000s
I20250905 19:59:45.828565 29763 raft_consensus.cc:513] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:45.727540 29931 consensus_queue.cc:1035] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.083s
I20250905 19:59:45.831043 29763 leader_election.cc:290] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:45.833283 28004 raft_consensus.cc:2391] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 4: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:45.834897 28076 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1320) took 2946 ms. Trace:
I20250905 19:59:45.835494 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:45.836272 28750 raft_consensus.cc:3058] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:45.835319 28076 rpcz_store.cc:276] 0905 19:59:42.889032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.962235 (+2073203us) service_pool.cc:225] Handling call
0905 19:59:45.046180 (+ 83945us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:45.264365 (+218185us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:45.288485 (+ 24120us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:45.460888 (+172403us) log.cc:844] Serialized 26 byte log entry
0905 19:59:45.697107 (+236219us) raft_consensus.cc:1643] Marking committed up to 1
0905 19:59:45.697161 (+ 54us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:45.697242 (+ 81us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:45.834658 (+137416us) raft_consensus.cc:1683] finished
0905 19:59:45.834677 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:45.834835 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30922,"raft.queue_time_us":223801,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":85243,"thread_start_us":294514,"threads_started":2,"wal-append.queue_time_us":336906}
I20250905 19:59:45.837394 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:45.837627 29996 raft_consensus.cc:513] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:45.742473 30008 raft_consensus.cc:3058] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:45.827690 29989 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:45.826186 28084 raft_consensus.cc:2466] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 19:59:45.845203 28151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 1914) took 2318 ms. Trace:
I20250905 19:59:45.756506 29901 raft_consensus.cc:491] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:45.841305 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 80870.1us
I20250905 19:59:45.756654 29936 raft_consensus.cc:2802] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:59:45.855021 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:45.855890 28750 raft_consensus.cc:2408] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250905 19:59:45.842415 28002 raft_consensus.cc:2466] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
W20250905 19:59:45.856982 28002 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1269) took 1477 ms (client timeout 1944 ms). Trace:
I20250905 19:59:45.802834 29853 raft_consensus.cc:2747] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:45.802615 29813 raft_consensus.cc:2747] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250905 19:59:45.857481 28002 rpcz_store.cc:269] 0905 19:59:44.379254 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.601837 (+1222583us) service_pool.cc:225] Handling call
0905 19:59:45.856926 (+255089us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:45.843935 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:45.860776 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 19:59:45.861250 28051 leader_election.cc:336] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.647s (SENT)
I20250905 19:59:45.846290 28079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1329) took 1776 ms. Trace:
I20250905 19:59:45.838610 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 19:59:45.862033 28051 leader_election.cc:336] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.579s (SENT)
I20250905 19:59:45.848956 28151 rpcz_store.cc:276] 0905 19:59:43.526988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.291691 (+1764703us) service_pool.cc:225] Handling call
0905 19:59:45.349375 (+ 57684us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:45.349639 (+ 264us) raft_consensus.cc:1532] Early marking committed up to index 3
0905 19:59:45.549013 (+199374us) log.cc:844] Serialized 14 byte log entry
0905 19:59:45.844877 (+295864us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:45.844918 (+ 41us) raft_consensus.cc:1643] Marking committed up to 3
0905 19:59:45.844953 (+ 35us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:45.845015 (+ 62us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:45.845142 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":461878,"raft.queue_time_us":96,"thread_start_us":433257,"threads_started":2}
I20250905 19:59:45.857074 28050 leader_election.cc:304] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
W20250905 19:59:45.839224 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 91301.6us
W20250905 19:59:45.865058 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:45.879452 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 19:59:45.851128 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 107063us
I20250905 19:59:45.881013 27973 leader_election.cc:304] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:45.881767 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:45.740031 29987 raft_consensus.cc:513] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:45.759164 29644 consensus_queue.cc:1035] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.093s
I20250905 19:59:45.884033 28003 raft_consensus.cc:2366] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a for earlier term 3. Current term is 4.
I20250905 19:59:45.886965 28125 leader_election.cc:304] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:45.838740 29906 raft_consensus.cc:513] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:45.855386 29936 raft_consensus.cc:491] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:45.801152 29840 consensus_queue.cc:1035] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20250905 19:59:45.882815 28745 raft_consensus.cc:491] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 19:59:45.846119 29809 raft_consensus.cc:513] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:45.848459 29931 raft_consensus.cc:2747] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:45.899207 27973 leader_election.cc:304] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:45.887943 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:45.858307 29881 raft_consensus.cc:2802] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:45.903712 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:45.803678 30000 consensus_queue.cc:1035] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.139s
I20250905 19:59:45.862403 28079 rpcz_store.cc:276] 0905 19:59:44.069900 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.707053 (+1637153us) service_pool.cc:225] Handling call
0905 19:59:45.707380 (+ 327us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:45.707574 (+ 194us) raft_consensus.cc:1532] Early marking committed up to index 3
0905 19:59:45.708288 (+ 714us) log.cc:844] Serialized 14 byte log entry
0905 19:59:45.846036 (+137748us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:45.846089 (+ 53us) raft_consensus.cc:1643] Marking committed up to 3
0905 19:59:45.846119 (+ 30us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:45.846163 (+ 44us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:45.846247 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":230,"prepare.queue_time_us":305,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":320,"spinlock_wait_cycles":9312512,"thread_start_us":137166,"threads_started":1}
I20250905 19:59:45.939162 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:45.939628 28747 raft_consensus.cc:1238] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:45.940896 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:45.941504 30000 consensus_queue.cc:1046] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250905 19:59:45.942811 30000 raft_consensus.cc:3053] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:45.943189 30000 raft_consensus.cc:738] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:45.943776 29840 raft_consensus.cc:491] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:45.944454 29840 raft_consensus.cc:513] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:45.946134 29997 raft_consensus.cc:513] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:45.948417 29997 leader_election.cc:290] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:45.947834 29840 leader_election.cc:290] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:44.859777 27975 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:54156 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1241)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1244)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1252)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1873)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1877)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1856)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1245)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1878)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1869)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 188)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1246)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1866)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 189)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 187)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1870)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 182)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1256)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1259)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 181)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1263)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 190)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1862)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1879)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1880)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1269)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1275)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1270)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1266)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1881)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1272)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1865)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1273)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1882)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1274)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1867)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1851)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1238)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1843)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1847)
I20250905 19:59:45.926263 29881 raft_consensus.cc:491] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:45.956851 29881 raft_consensus.cc:3058] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:45.860231 29901 raft_consensus.cc:513] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:45.900686 29943 raft_consensus.cc:2802] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election won for term 4
W20250905 19:59:45.920706 29894 consensus_meta.cc:287] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.858s user 0.006s sys 0.006s
I20250905 19:59:45.882087 29950 raft_consensus.cc:491] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:45.912760 29987 leader_election.cc:290] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:45.916162 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:45.839506 29941 raft_consensus.cc:513] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:45.847970 29934 raft_consensus.cc:2802] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:45.920888 28125 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:45.897359 29511 raft_consensus.cc:2747] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250905 19:59:45.922711 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:45.924742 28005 raft_consensus.cc:2391] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 4: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
W20250905 19:59:45.924741 28051 leader_election.cc:336] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.963s (SENT)
I20250905 19:59:45.928527 28718 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:45.964730 28051 leader_election.cc:304] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:45.929098 30017 raft_consensus.cc:2802] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 19:59:45.923760 29848 raft_consensus.cc:2747] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250905 19:59:45.929729 29989 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:45.932122 29906 leader_election.cc:290] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:45.921164 28745 raft_consensus.cc:3058] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:45.934931 28008 raft_consensus.cc:2391] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
W20250905 19:59:45.974299 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 53891.1us
I20250905 19:59:45.957863 29943 raft_consensus.cc:695] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:45.909415 28076 raft_consensus.cc:2391] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 4: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:45.944002 30000 consensus_queue.cc:260] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:45.979646 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:45.908443 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:45.962919 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:45.941749 28079 raft_consensus.cc:2466] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 19:59:45.980681 28151 raft_consensus.cc:2466] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 19:59:45.980854 28002 raft_consensus.cc:2466] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 3.
I20250905 19:59:45.919925 29936 raft_consensus.cc:3058] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:45.971802 28718 leader_election.cc:400] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 election: Vote denied by peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421) with higher term. Message: Invalid argument: T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a for earlier term 3. Current term is 4.
I20250905 19:59:45.962944 29950 raft_consensus.cc:3058] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:45.971012 29934 raft_consensus.cc:491] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:45.973727 28082 raft_consensus.cc:1238] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:45.901988 29903 consensus_queue.cc:1046] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.064s
W20250905 19:59:45.956962 28009 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1861) took 2434 ms (client timeout 1603 ms). Trace:
I20250905 19:59:45.505393 28746 rpcz_store.cc:276] 0905 19:59:43.404318 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:43.405394 (+ 1076us) service_pool.cc:225] Handling call
0905 19:59:44.213205 (+807811us) spinlock_profiling.cc:243] Waited 4.37 ms on lock 0x7b5800304598. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac6721e 000000000035a3c9 00007fe4a2b08310 00007fe4a2bceed4 00007fe4a2bcef41 00007fe4a2b91f60 00007fe4a2b7d1a9 00007fe4a2b7cc52 00007fe49c202b44
0905 19:59:44.213378 (+ 173us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:44.213647 (+ 269us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:44.213677 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:44.214162 (+ 485us) log.cc:844] Serialized 26 byte log entry
0905 19:59:44.898896 (+684734us) raft_consensus.cc:1643] Marking committed up to 4
0905 19:59:44.901151 (+ 2255us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:44.926411 (+ 25260us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:45.305479 (+379068us) raft_consensus.cc:1683] finished
0905 19:59:45.314020 (+ 8541us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:45.447342 (+133322us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10162560,"thread_start_us":640574,"threads_started":1,"wal-append.queue_time_us":624984}
I20250905 19:59:45.975207 28125 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:45.971359 30017 raft_consensus.cc:491] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:45.971359 29931 raft_consensus.cc:2747] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:45.983089 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:45.983652 28718 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:45.985000 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:45.977397 29992 raft_consensus.cc:2747] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250905 19:59:45.987365 28009 rpcz_store.cc:269] 0905 19:59:43.522748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.430393 (+907645us) service_pool.cc:225] Handling call
0905 19:59:45.956608 (+1526215us) spinlock_profiling.cc:243] Waited 393 ms on lock 0x7b5400291b54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:45.956842 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":904652672}
I20250905 19:59:45.976809 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:45.990919 28003 raft_consensus.cc:3058] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
W20250905 19:59:45.985527 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:45.993880 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:45.994012 29989 raft_consensus.cc:2747] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250905 19:59:45.994509 28009 raft_consensus.cc:3058] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:45.995463 29809 leader_election.cc:290] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:45.993360 28005 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:59:45.991711 28718 leader_election.cc:304] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:45.996683 29989 raft_consensus.cc:2802] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:45.997043 28084 raft_consensus.cc:2408] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250905 19:59:45.997069 29989 raft_consensus.cc:491] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:45.997677 29989 raft_consensus.cc:3058] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:45.996124 28005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1851) took 3110 ms. Trace:
I20250905 19:59:45.999588 28005 rpcz_store.cc:276] 0905 19:59:42.885699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.976471 (+3090772us) service_pool.cc:225] Handling call
0905 19:59:45.996068 (+ 19597us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:46.000504 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:46.000972 28003 raft_consensus.cc:2466] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 2.
I20250905 19:59:46.001238 28005 raft_consensus.cc:2391] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:46.001994 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:45.983173 29934 raft_consensus.cc:3058] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:46.002633 29824 raft_consensus.cc:3053] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:46.003039 29824 raft_consensus.cc:738] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:46.003290 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:46.003949 28003 raft_consensus.cc:2466] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 3.
I20250905 19:59:46.003933 29824 consensus_queue.cc:260] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:46.005057 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:46.005805 28003 raft_consensus.cc:2391] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:46.006740 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:46.002049 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:46.007171 28003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1238) took 3617 ms. Trace:
I20250905 19:59:46.007453 28002 raft_consensus.cc:2366] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 2. Current term is 3.
I20250905 19:59:46.007524 28003 rpcz_store.cc:276] 0905 19:59:42.389607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:46.006336 (+3616729us) service_pool.cc:225] Handling call
0905 19:59:46.006687 (+ 351us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:46.006844 (+ 157us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:46.006869 (+ 25us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:46.006915 (+ 46us) raft_consensus.cc:1643] Marking committed up to 5
0905 19:59:46.006947 (+ 32us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:46.006998 (+ 51us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:46.007115 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:45.974982 29840 raft_consensus.cc:2747] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250905 19:59:46.009334 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:45.992672 28125 leader_election.cc:304] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:46.010839 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:46.015236 28718 leader_election.cc:304] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:46.015450 28125 leader_election.cc:304] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:46.016466 29933 raft_consensus.cc:2762] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250905 19:59:46.016347 28125 leader_election.cc:304] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:46.017277 29987 raft_consensus.cc:2802] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:46.017899 29987 raft_consensus.cc:695] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:46.017968 29933 raft_consensus.cc:2802] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:46.018718 29933 raft_consensus.cc:695] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:46.019033 29987 consensus_queue.cc:237] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:46.019660 29933 consensus_queue.cc:237] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:45.990509 30017 raft_consensus.cc:3058] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:46.022841 30000 raft_consensus.cc:3058] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:46.022738 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:46.024272 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:46.025092 28076 raft_consensus.cc:2391] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 3: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:45.979032 29928 raft_consensus.cc:2802] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:46.032639 28125 leader_election.cc:304] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:46.033387 28151 raft_consensus.cc:2466] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:46.005347 29824 raft_consensus.cc:3058] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:45.972098 29894 raft_consensus.cc:513] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:46.037067 28718 leader_election.cc:304] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:46.033771 28009 raft_consensus.cc:2466] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:46.035391 28050 leader_election.cc:304] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:46.012574 29848 raft_consensus.cc:2802] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:46.028980 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:46.035912 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:46.041949 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:46.041999 28718 leader_election.cc:304] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:46.036322 29854 raft_consensus.cc:2747] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 19:59:46.046833 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:46.028255 29809 raft_consensus.cc:2747] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:46.047505 28079 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6.
I20250905 19:59:46.043411 29992 raft_consensus.cc:2747] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:46.048215 29916 raft_consensus.cc:2802] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:46.039198 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:46.044950 28009 raft_consensus.cc:3058] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:46.040295 29928 raft_consensus.cc:695] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:46.061533 28717 leader_election.cc:304] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
W20250905 19:59:46.036101 29985 log.cc:927] Time spent T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 1.121s user 0.004s sys 0.000s
I20250905 19:59:46.049885 29644 raft_consensus.cc:2802] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:46.045789 28003 raft_consensus.cc:1238] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20250905 19:59:46.091372 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:46.093272 28717 leader_election.cc:304] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:46.046736 29941 leader_election.cc:290] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:46.100574 28156 raft_consensus.cc:2466] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:46.116134 28050 leader_election.cc:304] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:46.043916 29848 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:46.103830 28003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1843) took 3480 ms. Trace:
I20250905 19:59:46.142578 28003 rpcz_store.cc:276] 0905 19:59:42.623837 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:46.027830 (+3403993us) service_pool.cc:225] Handling call
0905 19:59:46.045567 (+ 17737us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:46.103546 (+ 57979us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:46.103763 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:46.104010 28717 leader_election.cc:304] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:46.052531 28715 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:46.061784 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
W20250905 19:59:46.144511 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 92068us
I20250905 19:59:46.106848 28079 raft_consensus.cc:1273] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250905 19:59:46.052829 29854 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:46.010375 30002 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:46.053148 29878 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:46.112659 29840 raft_consensus.cc:2802] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 19:59:46.055682 29996 leader_election.cc:290] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:46.114149 27973 leader_election.cc:304] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:46.152644 28009 raft_consensus.cc:2466] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 3.
I20250905 19:59:46.055094 29916 raft_consensus.cc:491] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:46.113449 29644 raft_consensus.cc:695] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:46.121337 29995 raft_consensus.cc:3053] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Stepping down as leader of term 1
W20250905 19:59:46.122283 28006 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1275) took 1352 ms (client timeout 1630 ms). Trace:
I20250905 19:59:46.122628 30004 raft_consensus.cc:2802] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:46.128343 29900 raft_consensus.cc:3053] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:46.091390 29809 raft_consensus.cc:491] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:46.035218 29943 consensus_queue.cc:237] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:46.169113 28159 raft_consensus.cc:1238] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250905 19:59:46.151963 29840 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:46.131520 29881 raft_consensus.cc:513] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:46.133060 29950 raft_consensus.cc:513] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:46.102789 29992 raft_consensus.cc:2747] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 19:59:46.136011 29894 leader_election.cc:290] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:46.136373 29864 leader_election.cc:290] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:46.137828 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 19:59:46.177738 28001 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 180) took 2750 ms (client timeout 1855 ms). Trace:
I20250905 19:59:46.146453 28079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 177) took 1969 ms. Trace:
I20250905 19:59:46.178755 28079 rpcz_store.cc:276] 0905 19:59:44.177455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:46.053795 (+1876340us) service_pool.cc:225] Handling call
0905 19:59:46.101025 (+ 47230us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:46.146086 (+ 45061us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:46.146344 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:46.178308 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:59:46.178155 28001 rpcz_store.cc:269] 0905 19:59:43.426883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:46.177544 (+2750661us) service_pool.cc:212] Skipping call since client already timed out
0905 19:59:46.177664 (+ 120us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250905 19:59:46.181023 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:46.181653 28156 raft_consensus.cc:3058] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:46.147059 29848 raft_consensus.cc:3058] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:46.148386 28076 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1346) took 1914 ms. Trace:
W20250905 19:59:46.143919 28051 leader_election.cc:336] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 2.168s (SENT)
W20250905 19:59:46.150357 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 46411.7us
I20250905 19:59:46.166023 29900 raft_consensus.cc:738] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:46.151973 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:46.155920 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:46.156101 28718 leader_election.cc:304] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:46.120594 29508 leader_election.cc:290] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:46.185716 28051 leader_election.cc:304] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:46.187175 28717 leader_election.cc:304] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:46.161578 28750 raft_consensus.cc:1238] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:46.153141 29801 raft_consensus.cc:2747] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:46.163141 29928 consensus_queue.cc:237] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:46.163866 28154 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39586 (request call id 178) took 2704 ms (client timeout 1680 ms). Trace:
W20250905 19:59:46.190133 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 46288.4us
W20250905 19:59:46.190726 28154 rpcz_store.cc:269] 0905 19:59:43.462258 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:43.852059 (+389801us) service_pool.cc:225] Handling call
0905 19:59:46.163626 (+2311567us) spinlock_profiling.cc:243] Waited 1.6 s on lock 0x7b5400186354. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:46.163779 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3682791808}
I20250905 19:59:46.164794 28752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40904 (request call id 502) took 1039 ms. Trace:
I20250905 19:59:46.152580 29813 raft_consensus.cc:2802] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250905 19:59:46.131426 28150 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39586 (request call id 179) took 2657 ms (client timeout 1811 ms). Trace:
W20250905 19:59:46.199792 28150 rpcz_store.cc:269] 0905 19:59:43.476626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.436067 (+959441us) service_pool.cc:225] Handling call
0905 19:59:46.131209 (+1695142us) spinlock_profiling.cc:243] Waited 68.6 ms on lock 0x7b5400200b54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:46.131338 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":157850368}
I20250905 19:59:46.201442 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:46.157240 29763 raft_consensus.cc:2747] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250905 19:59:46.172187 27975 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:54156 dropped due to backpressure: service queue is full with 50 items [suppressed 1 similar messages]
I20250905 19:59:46.158763 29916 raft_consensus.cc:3058] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:46.185554 28076 rpcz_store.cc:276] 0905 19:59:44.233874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:46.112795 (+1878921us) service_pool.cc:225] Handling call
0905 19:59:46.113222 (+ 427us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:46.147782 (+ 34560us) raft_consensus.cc:1532] Early marking committed up to index 3
0905 19:59:46.147816 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:46.147892 (+ 76us) raft_consensus.cc:1643] Marking committed up to 3
0905 19:59:46.147971 (+ 79us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:46.148085 (+ 114us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:46.148289 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29794,"raft.queue_time_us":28178,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":2820}
W20250905 19:59:46.208541 28083 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1336) took 2136 ms (client timeout 2196 ms). Trace:
W20250905 19:59:46.208976 28083 rpcz_store.cc:269] 0905 19:59:44.071696 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.500207 (+428511us) service_pool.cc:225] Handling call
0905 19:59:46.208308 (+1708101us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54003aa054. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:46.208469 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2540821248}
I20250905 19:59:46.209772 28076 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1352) took 1336 ms. Trace:
I20250905 19:59:46.211083 29917 raft_consensus.cc:3053] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:46.211526 29917 raft_consensus.cc:738] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:46.212003 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:46.212628 28154 raft_consensus.cc:3058] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:46.212390 29917 consensus_queue.cc:260] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:46.213577 29917 raft_consensus.cc:3058] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:46.214735 29801 consensus_queue.cc:1035] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250905 19:59:46.217834 29950 leader_election.cc:290] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:46.222121 29916 raft_consensus.cc:513] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:46.223801 29996 raft_consensus.cc:2747] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 19:59:46.224435 29881 leader_election.cc:290] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:46.225741 29644 consensus_queue.cc:237] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:46.231498 28156 raft_consensus.cc:2466] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:46.235035 28050 leader_election.cc:304] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:46.175897 28049 leader_election.cc:304] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:46.161417 29995 raft_consensus.cc:738] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:46.150668 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:46.186745 28009 raft_consensus.cc:2391] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
W20250905 19:59:46.159180 28006 rpcz_store.cc:269] 0905 19:59:44.769763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.657841 (+888078us) service_pool.cc:225] Handling call
0905 19:59:46.122087 (+464246us) spinlock_profiling.cc:243] Waited 61.9 ms on lock 0x7b540016e9d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:46.122222 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":142465792}
I20250905 19:59:46.189334 28005 raft_consensus.cc:2391] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 4: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:46.152877 29854 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:46.192332 28752 rpcz_store.cc:276] 0905 19:59:45.125046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.137752 (+ 12706us) service_pool.cc:225] Handling call
0905 19:59:45.159705 (+ 21953us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:45.820522 (+660817us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:46.164567 (+344045us) spinlock_profiling.cc:243] Waited 585 ms on lock 0x7b54004b0d54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:46.164733 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1346201344}
I20250905 19:59:46.167735 29901 leader_election.cc:290] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:46.154502 29878 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:46.175604 29840 raft_consensus.cc:3058] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:46.244354 30002 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:46.206312 27975 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:54156 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1892)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 205)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1291)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1891)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1895)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1910)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1890)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1889)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 203)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1906)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1894)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1292)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1920)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1918)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1276)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1279)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 193)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 199)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1283)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1284)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1285)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1286)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1287)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1288)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1289)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1290)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1886)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1893)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1903)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1909)
I20250905 19:59:46.198392 28079 raft_consensus.cc:1238] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:46.247062 28079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1350) took 1386 ms. Trace:
I20250905 19:59:46.247430 28079 rpcz_store.cc:276] 0905 19:59:44.860611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:46.179367 (+1318756us) service_pool.cc:225] Handling call
0905 19:59:46.198196 (+ 18829us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:46.246787 (+ 48591us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:46.247004 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:46.248503 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:46.198122 29900 consensus_queue.cc:260] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:46.178288 30004 raft_consensus.cc:695] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:46.207214 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:59:46.236310 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 60567.5us
W20250905 19:59:46.241421 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:46.249297 28003 raft_consensus.cc:2374] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3. Re-sending same reply.
I20250905 19:59:46.268437 28001 raft_consensus.cc:2391] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:46.257575 28154 raft_consensus.cc:2466] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 2.
I20250905 19:59:46.247164 29995 consensus_queue.cc:260] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:46.269872 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:46.172201 29809 raft_consensus.cc:513] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:46.200345 29931 raft_consensus.cc:2747] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:46.280814 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:46.199648 29813 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250905 19:59:46.287171 27973 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:46.292068 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:46.294857 29813 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Advancing to term 7
I20250905 19:59:46.297756 30017 raft_consensus.cc:513] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:46.253085 29941 raft_consensus.cc:2802] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:46.262898 29826 raft_consensus.cc:2747] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 19:59:46.304409 29840 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:46.310454 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 85473.1us
I20250905 19:59:46.311545 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:46.312352 28003 raft_consensus.cc:2466] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
I20250905 19:59:46.310352 28154 raft_consensus.cc:1238] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250905 19:59:46.308586 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:46.338323 28750 raft_consensus.cc:3058] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Advancing to term 3
W20250905 19:59:46.312039 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 124940us
I20250905 19:59:46.343744 28083 raft_consensus.cc:1238] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20250905 19:59:46.345047 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:46.345731 28717 leader_election.cc:304] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:46.345984 28083 raft_consensus.cc:3058] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
W20250905 19:59:46.313830 28051 leader_election.cc:336] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 2.111s (SENT)
I20250905 19:59:46.350343 28752 raft_consensus.cc:1238] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 LEADER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:46.309067 30008 raft_consensus.cc:2747] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:46.352720 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:46.353204 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:46.351261 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:46.354868 28001 raft_consensus.cc:2366] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4.
I20250905 19:59:46.300365 29995 raft_consensus.cc:3058] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:46.365348 28083 raft_consensus.cc:2466] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
W20250905 19:59:46.366837 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:46.367544 28083 raft_consensus.cc:1238] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:46.372629 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:46.372992 28750 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:46.368450 28717 leader_election.cc:304] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 19:59:46.375385 28751 raft_consensus.cc:2408] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250905 19:59:46.375890 27974 leader_election.cc:304] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:46.343050 29941 raft_consensus.cc:695] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:46.320264 30017 leader_election.cc:290] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 19:59:46.379849 30034 log.cc:927] Time spent T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.081s user 0.001s sys 0.000s
I20250905 19:59:46.379635 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:46.381227 28746 raft_consensus.cc:1238] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 4. Current term is 5. Ops: []
W20250905 19:59:46.381750 30010 log.cc:927] Time spent T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.811s user 0.002s sys 0.000s
I20250905 19:59:46.210249 28076 rpcz_store.cc:276] 0905 19:59:44.873637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:46.208677 (+1335040us) service_pool.cc:225] Handling call
0905 19:59:46.209037 (+ 360us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:46.209370 (+ 333us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:46.209411 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:46.209460 (+ 49us) raft_consensus.cc:1643] Marking committed up to 1
0905 19:59:46.209503 (+ 43us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:46.209579 (+ 76us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:46.209709 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7229,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":112}
I20250905 19:59:46.391585 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:46.386440 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:46.337664 29864 consensus_queue.cc:1046] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.035s
I20250905 19:59:46.387555 28001 raft_consensus.cc:2366] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4.
I20250905 19:59:46.309366 29989 raft_consensus.cc:513] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:46.393309 28076 raft_consensus.cc:1273] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250905 19:59:46.398459 28079 raft_consensus.cc:1238] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250905 19:59:46.369977 29511 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:46.406497 29989 leader_election.cc:290] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:46.362695 29931 consensus_queue.cc:1046] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20250905 19:59:46.352312 29881 consensus_queue.cc:1046] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20250905 19:59:46.372114 30037 raft_consensus.cc:2747] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:46.351416 29813 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:46.410825 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:46.414361 28003 raft_consensus.cc:3058] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:46.394681 29763 raft_consensus.cc:2747] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250905 19:59:46.383584 30036 raft_consensus.cc:2802] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:59:46.412812 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:59:46.408305 30005 consensus_meta.cc:287] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.706s user 0.010s sys 0.001s
I20250905 19:59:46.424276 28002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1847) took 3673 ms. Trace:
I20250905 19:59:46.394059 29848 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:46.426887 28002 rpcz_store.cc:276] 0905 19:59:42.751125 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:46.026846 (+3275721us) service_pool.cc:225] Handling call
0905 19:59:46.041826 (+ 14980us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:46.042180 (+ 354us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:46.042207 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:46.053958 (+ 11751us) log.cc:844] Serialized 26 byte log entry
0905 19:59:46.236258 (+182300us) raft_consensus.cc:1643] Marking committed up to 2
0905 19:59:46.236296 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:46.236358 (+ 62us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:46.424058 (+187700us) raft_consensus.cc:1683] finished
0905 19:59:46.424080 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:46.424214 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1291,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":105,"thread_start_us":53592,"threads_started":1,"wal-append.queue_time_us":195574}
I20250905 19:59:46.445518 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:46.383353 30004 consensus_queue.cc:237] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:46.407984 29511 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:46.414928 29989 consensus_queue.cc:1046] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20250905 19:59:46.507601 28078 raft_consensus.cc:1238] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 3. Current term is 4. Ops: []
W20250905 19:59:46.508427 28051 leader_election.cc:336] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.942s (SENT)
I20250905 19:59:46.421648 29916 leader_election.cc:290] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:46.529495 28007 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1853) took 3402 ms (client timeout 1963 ms). Trace:
W20250905 19:59:46.529986 28007 rpcz_store.cc:269] 0905 19:59:43.126835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.150592 (+1023757us) service_pool.cc:225] Handling call
0905 19:59:46.529280 (+2378688us) spinlock_profiling.cc:243] Waited 572 ms on lock 0x7b54000cb054. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:46.529414 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1314837248}
I20250905 19:59:46.531037 28082 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1325) took 3265 ms. Trace:
I20250905 19:59:46.531211 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:59:46.531610 28008 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1862) took 3003 ms (client timeout 2687 ms). Trace:
I20250905 19:59:46.531509 28082 rpcz_store.cc:276] 0905 19:59:43.265743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.200096 (+1934353us) service_pool.cc:225] Handling call
0905 19:59:45.201446 (+ 1350us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:45.982959 (+781513us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:46.530773 (+547814us) spinlock_profiling.cc:243] Waited 772 ms on lock 0x7b540029ac54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:46.530978 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1775543168}
I20250905 19:59:46.532418 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:46.533154 28007 raft_consensus.cc:2391] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
W20250905 19:59:46.529498 28153 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 1919) took 2819 ms (client timeout 2182 ms). Trace:
W20250905 19:59:46.479583 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:46.532433 28008 rpcz_store.cc:269] 0905 19:59:43.528018 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.509383 (+1981365us) service_pool.cc:225] Handling call
0905 19:59:46.531333 (+1021950us) spinlock_profiling.cc:243] Waited 424 ms on lock 0x7b54001fb654. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:46.531517 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":975332096}
I20250905 19:59:46.470983 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:46.472869 29854 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:46.417569 29881 consensus_queue.cc:1230] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer 7e9c9f591f05464f9b097bf003d14f4a log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20250905 19:59:46.522897 28077 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1330) took 2452 ms (client timeout 1877 ms). Trace:
I20250905 19:59:46.484630 29878 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 19:59:46.539705 28077 rpcz_store.cc:269] 0905 19:59:44.070204 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.217321 (+147117us) service_pool.cc:225] Handling call
0905 19:59:46.505288 (+2287967us) spinlock_profiling.cc:243] Waited 564 ms on lock 0x7b540036e054. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:46.522813 (+ 17525us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1298034944}
W20250905 19:59:46.540711 30040 log.cc:927] Time spent T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.084s user 0.000s sys 0.001s
I20250905 19:59:46.473587 28154 raft_consensus.cc:2391] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
W20250905 19:59:46.523881 28745 consensus_meta.cc:287] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.540s user 0.011s sys 0.000s
I20250905 19:59:46.480899 29809 leader_election.cc:290] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:46.526798 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:46.543134 28746 raft_consensus.cc:2466] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 19:59:46.543272 28745 raft_consensus.cc:513] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:46.543838 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:46.534690 28051 leader_election.cc:304] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 19:59:46.534264 28153 rpcz_store.cc:269] 0905 19:59:43.710165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.856781 (+1146616us) service_pool.cc:225] Handling call
0905 19:59:46.529305 (+1672524us) spinlock_profiling.cc:243] Waited 48.3 ms on lock 0x7b5400326c54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:46.529405 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":111128960}
I20250905 19:59:46.544775 29878 raft_consensus.cc:2747] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 19:59:46.545029 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:46.545161 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:46.545616 28150 raft_consensus.cc:3058] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:46.546408 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:46.547173 28153 raft_consensus.cc:2466] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
I20250905 19:59:46.547328 29894 raft_consensus.cc:2762] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20250905 19:59:46.548902 28050 leader_election.cc:304] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:46.549360 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:46.549716 29931 raft_consensus.cc:2747] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 19:59:46.550115 28752 raft_consensus.cc:2466] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
I20250905 19:59:46.536554 28003 raft_consensus.cc:2466] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 19:59:46.482277 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:46.470937 28750 raft_consensus.cc:1238] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 3. Current term is 4. Ops: []
W20250905 19:59:46.506705 29934 consensus_meta.cc:287] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.504s user 0.009s sys 0.002s
W20250905 19:59:46.540799 29936 consensus_meta.cc:287] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.551s user 0.011s sys 0.000s
I20250905 19:59:46.521543 29840 leader_election.cc:290] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:46.539357 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:46.561264 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:46.562224 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:46.445775 29987 consensus_queue.cc:1035] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.097s
W20250905 19:59:46.484835 30033 log.cc:927] Time spent T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.141s user 0.000s sys 0.001s
I20250905 19:59:46.554296 28050 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:46.555245 28001 raft_consensus.cc:2466] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:46.555078 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:46.550868 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:46.566763 28008 raft_consensus.cc:2391] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:46.424513 29824 consensus_queue.cc:1046] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.138s
I20250905 19:59:46.563423 28003 raft_consensus.cc:2391] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:46.569175 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:46.447005 30036 raft_consensus.cc:695] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 2 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:46.541404 29848 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 19:59:46.572672 28085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1332) took 2501 ms (client timeout 1722 ms). Trace:
I20250905 19:59:46.572957 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:59:46.568676 28004 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 187) took 2894 ms (client timeout 2128 ms). Trace:
I20250905 19:59:46.562258 28123 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 19:59:46.570858 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:46.447300 30005 raft_consensus.cc:513] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:46.563519 28007 raft_consensus.cc:2408] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250905 19:59:46.571463 28150 raft_consensus.cc:2466] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
W20250905 19:59:46.573112 28085 rpcz_store.cc:269] 0905 19:59:44.070733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.154058 (+ 83325us) service_pool.cc:225] Handling call
0905 19:59:46.572462 (+2418404us) spinlock_profiling.cc:243] Waited 310 ms on lock 0x7b540019ebd4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:46.572597 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":713899264}
I20250905 19:59:46.568156 28752 raft_consensus.cc:2391] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 3: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
W20250905 19:59:46.573976 28004 rpcz_store.cc:269] 0905 19:59:43.674123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.007956 (+1333833us) service_pool.cc:225] Handling call
0905 19:59:46.568445 (+1560489us) spinlock_profiling.cc:243] Waited 667 ms on lock 0x7b54003840d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:46.568607 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1535021824}
I20250905 19:59:46.587472 29813 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:46.588148 29644 raft_consensus.cc:3053] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:59:46.587982 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:46.588647 29644 raft_consensus.cc:738] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:46.569228 29878 raft_consensus.cc:2802] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 19:59:46.590200 29878 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:46.575100 28002 raft_consensus.cc:2374] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3. Re-sending same reply.
I20250905 19:59:46.580092 28748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40926 (request call id 386) took 3071 ms. Trace:
I20250905 19:59:46.591190 28748 rpcz_store.cc:276] 0905 19:59:43.508511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.593582 (+1085071us) service_pool.cc:225] Handling call
0905 19:59:44.594050 (+ 468us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:45.548030 (+953980us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:46.579800 (+1031770us) spinlock_profiling.cc:243] Waited 620 ms on lock 0x7b54005302d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:46.580014 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1425527040}
W20250905 19:59:46.592321 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 1.3497e+06us
I20250905 19:59:46.580134 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:46.585369 28157 raft_consensus.cc:1238] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 4. Current term is 5. Ops: []
W20250905 19:59:46.585254 28155 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 1922) took 2432 ms (client timeout 2271 ms). Trace:
W20250905 19:59:46.593127 28155 rpcz_store.cc:269] 0905 19:59:44.152463 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.885530 (+733067us) service_pool.cc:225] Handling call
0905 19:59:46.585038 (+1699508us) spinlock_profiling.cc:243] Waited 677 ms on lock 0x7b540009e7d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:46.585178 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1556590848}
I20250905 19:59:46.595191 28077 raft_consensus.cc:1273] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250905 19:59:46.595234 28157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39586 (request call id 190) took 2412 ms. Trace:
W20250905 19:59:46.596066 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:46.596731 28008 raft_consensus.cc:2391] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
W20250905 19:59:46.586143 28010 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1845) took 3956 ms (client timeout 1969 ms). Trace:
I20250905 19:59:46.562971 29936 raft_consensus.cc:513] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:46.590620 29878 raft_consensus.cc:3058] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:46.597057 30032 raft_consensus.cc:2802] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 19:59:46.576701 29809 consensus_queue.cc:1046] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250905 19:59:46.563272 29934 raft_consensus.cc:513] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:46.596089 28157 rpcz_store.cc:276] 0905 19:59:44.182462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.390112 (+1207650us) service_pool.cc:225] Handling call
0905 19:59:46.585173 (+1195061us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:46.593090 (+ 7917us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:46.595024 (+ 1934us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b540009e7d0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:46.595171 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2610174720}
I20250905 19:59:46.598615 28078 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 172) took 2780 ms. Trace:
I20250905 19:59:46.740597 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:46.748698 28078 rpcz_store.cc:276] 0905 19:59:43.817741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.692868 (+1875127us) service_pool.cc:225] Handling call
0905 19:59:46.507404 (+814536us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:46.527832 (+ 20428us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:46.598280 (+ 70448us) spinlock_profiling.cc:243] Waited 814 ms on lock 0x7b540036e050. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:46.598507 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1872198912}
I20250905 19:59:46.750015 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 19:59:46.657086 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:46.462257 29941 consensus_queue.cc:237] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:46.711314 28159 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 1905) took 3855 ms. Trace:
I20250905 19:59:46.713107 29864 raft_consensus.cc:3053] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:59:46.750340 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:46.741171 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:46.752422 28004 raft_consensus.cc:3058] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:46.752332 28159 rpcz_store.cc:276] 0905 19:59:42.855693 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.969377 (+2113684us) service_pool.cc:225] Handling call
0905 19:59:46.165155 (+1195778us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:46.169566 (+ 4411us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:46.711088 (+541522us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b5400186350. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:46.711250 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2749169920}
I20250905 19:59:46.753508 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:46.754257 28078 raft_consensus.cc:2391] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 4: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:46.712919 30036 consensus_queue.cc:237] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:46.714516 28744 raft_consensus.cc:1238] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:46.715283 30005 leader_election.cc:290] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:46.715970 30029 log.cc:927] Time spent T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.574s user 0.000s sys 0.009s
I20250905 19:59:46.742750 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:46.716032 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:46.775379 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:46.775878 28152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 1936) took 1030 ms. Trace:
I20250905 19:59:46.776311 28005 raft_consensus.cc:2391] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:46.777397 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:46.776366 28152 rpcz_store.cc:276] 0905 19:59:45.745906 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.746025 (+ 119us) service_pool.cc:225] Handling call
0905 19:59:45.802125 (+ 56100us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:45.802598 (+ 473us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:45.802641 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:45.840229 (+ 37588us) log.cc:844] Serialized 26 byte log entry
0905 19:59:45.978395 (+138166us) raft_consensus.cc:1643] Marking committed up to 2
0905 19:59:45.978468 (+ 73us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:45.978535 (+ 67us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:46.675788 (+697253us) spinlock_profiling.cc:243] Waited 4.09 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49d85891e 00007fe49d857c66 00007fe49d857888 00007fe49d8612c1 00007fe49d861279 00007fe49d861209
0905 19:59:46.775501 (+ 99713us) raft_consensus.cc:1683] finished
0905 19:59:46.775528 (+ 27us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:46.775806 (+ 278us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2561,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":35187,"spinlock_wait_cycles":10077312,"thread_start_us":82352,"threads_started":1,"wal-append.queue_time_us":192319}
I20250905 19:59:46.778024 28005 raft_consensus.cc:2466] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
W20250905 19:59:46.717762 30009 log.cc:927] Time spent T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 1.134s user 0.006s sys 0.006s
I20250905 19:59:46.718819 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 19:59:46.718494 28153 raft_consensus.cc:1273] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250905 19:59:46.783989 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:59:46.719858 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 19:59:46.721164 27894 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet fc7dc431504743d79dbdc8eea83f9dcb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 26b2f874b8e54f08b01a9ed00d5cce05 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
W20250905 19:59:46.718531 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 162707us
W20250905 19:59:46.786832 27894 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 65725.2us
I20250905 19:59:46.589538 29644 consensus_queue.cc:260] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:46.724460 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:46.724602 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:46.802042 28001 raft_consensus.cc:2366] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 5.
I20250905 19:59:46.725325 28717 leader_election.cc:304] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 19:59:46.722338 28156 raft_consensus.cc:1238] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250905 19:59:46.802897 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:46.803666 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:46.743522 29936 leader_election.cc:290] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:46.729132 28083 raft_consensus.cc:1273] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:59:46.726681 28125 leader_election.cc:304] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:46.806264 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:46.806919 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 19:59:46.807193 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 80605.9us
I20250905 19:59:46.807343 28005 raft_consensus.cc:2391] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:46.803707 28001 raft_consensus.cc:2391] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 6: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:46.730396 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:46.808039 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:46.730648 28751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40918 (request call id 462) took 1020 ms. Trace:
W20250905 19:59:46.735534 27973 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:46.718896 28122 connection.cc:232] RPC call timeout handler was delayed by -1.0637853990000394s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19390000000000002s for a response
I20250905 19:59:46.738055 29992 consensus_queue.cc:1035] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20250905 19:59:46.738222 30032 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:46.816893 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:46.817812 28003 raft_consensus.cc:2391] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:46.745699 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:46.746316 28747 raft_consensus.cc:1238] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:46.747017 29809 raft_consensus.cc:3053] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:59:46.755729 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:46.821027 28007 raft_consensus.cc:2391] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:46.756173 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:46.821280 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:46.821671 28002 raft_consensus.cc:2391] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:46.718698 29900 raft_consensus.cc:3058] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:46.724313 30017 raft_consensus.cc:3053] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:46.760324 29994 consensus_queue.cc:1046] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250905 19:59:46.760625 27908 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34446 (request call id 238) took 7734 ms. Trace:
I20250905 19:59:46.742419 28743 raft_consensus.cc:1238] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:46.742192 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:46.834071 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:46.761752 27904 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34440 (request call id 252) took 7143 ms. Trace:
I20250905 19:59:46.755244 29864 raft_consensus.cc:738] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:46.743587 28745 leader_election.cc:290] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:46.742583 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:46.762789 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 19:59:46.763064 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:46.724071 29917 raft_consensus.cc:3053] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Stepping down as leader of term 4
I20250905 19:59:46.765280 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:46.779067 29511 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:46.676877 30037 raft_consensus.cc:3053] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Stepping down as leader of term 4
I20250905 19:59:46.742098 27907 catalog_manager.cc:5582] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 1 to 2, leader changed from cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 2 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20250905 19:59:46.859563 28003 raft_consensus.cc:1238] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:46.860261 28003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1279) took 1750 ms. Trace:
I20250905 19:59:46.859891 27907 catalog_manager.cc:5582] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 2 to 3, leader changed from cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 3 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20250905 19:59:46.860594 28003 rpcz_store.cc:276] 0905 19:59:45.109832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:46.831350 (+1721518us) service_pool.cc:225] Handling call
0905 19:59:46.859398 (+ 28048us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:46.860012 (+ 614us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:46.860205 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:46.825718 29809 raft_consensus.cc:738] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:46.720144 28155 raft_consensus.cc:1273] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250905 19:59:46.790900 28159 raft_consensus.cc:1238] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:46.798244 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:46.870690 28006 raft_consensus.cc:1238] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:46.871234 28006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1284) took 1201 ms. Trace:
I20250905 19:59:46.871519 28006 rpcz_store.cc:276] 0905 19:59:45.669529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:46.870183 (+1200654us) service_pool.cc:225] Handling call
0905 19:59:46.870503 (+ 320us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:46.871030 (+ 527us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:46.871179 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:46.799566 28151 raft_consensus.cc:1238] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:46.804829 28153 raft_consensus.cc:1238] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 2. Current term is 3. Ops: []
W20250905 19:59:46.805320 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 80072.9us
I20250905 19:59:46.873271 28004 raft_consensus.cc:2466] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
W20250905 19:59:46.734797 28010 rpcz_store.cc:269] 0905 19:59:42.629687 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:43.270128 (+640441us) service_pool.cc:225] Handling call
0905 19:59:46.585923 (+3315795us) spinlock_profiling.cc:243] Waited 1.73 s on lock 0x7b5400384ad4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:46.586050 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3983408896}
I20250905 19:59:46.810267 27973 leader_election.cc:304] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:46.811197 28751 rpcz_store.cc:276] 0905 19:59:45.710225 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.710458 (+ 233us) service_pool.cc:225] Handling call
0905 19:59:46.610002 (+899544us) spinlock_profiling.cc:243] Waited 554 ms on lock 0x7b540069c8d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:46.730394 (+120392us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1273878400}
I20250905 19:59:46.812686 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:46.812726 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:46.814675 28001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1276) took 1846 ms. Trace:
I20250905 19:59:46.875986 28004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1286) took 1205 ms. Trace:
I20250905 19:59:46.861963 27907 catalog_manager.cc:5582] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 2 to 3, leader changed from 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 3 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:46.876194 28001 rpcz_store.cc:276] 0905 19:59:44.968011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:46.813660 (+1845649us) service_pool.cc:225] Handling call
0905 19:59:46.813952 (+ 292us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:46.814279 (+ 327us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:46.814308 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:46.814379 (+ 71us) raft_consensus.cc:1643] Marking committed up to 1
0905 19:59:46.814412 (+ 33us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:46.814496 (+ 84us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:46.814617 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":82,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":13255}
I20250905 19:59:46.876355 28004 rpcz_store.cc:276] 0905 19:59:45.670148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:46.873925 (+1203777us) service_pool.cc:225] Handling call
0905 19:59:46.875489 (+ 1564us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:46.875664 (+ 175us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:46.875689 (+ 25us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:46.875730 (+ 41us) raft_consensus.cc:1643] Marking committed up to 3
0905 19:59:46.875765 (+ 35us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:46.875817 (+ 52us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:46.875931 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:46.878249 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:46.878786 28001 raft_consensus.cc:1273] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250905 19:59:46.879029 28153 raft_consensus.cc:2391] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 3: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:46.879454 28004 raft_consensus.cc:1273] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:59:46.879554 28001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1289) took 1156 ms. Trace:
I20250905 19:59:46.879848 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:46.880422 28153 raft_consensus.cc:3058] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:46.817090 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:46.817902 29924 consensus_queue.cc:1046] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20250905 19:59:46.775434 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:46.819703 28157 raft_consensus.cc:1238] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:46.822038 28156 raft_consensus.cc:2366] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6.
I20250905 19:59:46.829557 28007 raft_consensus.cc:1238] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 4. Current term is 5. Ops: []
I20250905 19:59:46.889436 28007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 193) took 1250 ms. Trace:
I20250905 19:59:46.761322 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 19:59:46.836349 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:46.756501 29871 consensus_queue.cc:1046] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.067s
I20250905 19:59:46.812465 29987 raft_consensus.cc:2802] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader election won for term 5
I20250905 19:59:46.844553 28005 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 1.
I20250905 19:59:46.847175 28752 raft_consensus.cc:1238] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 4. Ops: []
I20250905 19:59:46.819917 30032 raft_consensus.cc:3058] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:46.817435 29848 raft_consensus.cc:2802] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:46.851825 28008 raft_consensus.cc:2391] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 4: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:46.923061 28008 raft_consensus.cc:1273] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250905 19:59:46.844497 27904 rpcz_store.cc:276] 0905 19:59:39.605026 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:39.605192 (+ 166us) service_pool.cc:225] Handling call
0905 19:59:46.748037 (+7142845us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:59:42.190172 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:59:42.253423 (+ 63251us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:59:42.253479 (+ 56us) write_op.cc:435] Acquired schema lock
0905 19:59:42.253507 (+ 28us) tablet.cc:660] Decoding operations
0905 19:59:42.254233 (+ 726us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:59:42.254368 (+ 135us) write_op.cc:641] Partition lock acquired for write op
0905 19:59:42.254412 (+ 44us) tablet.cc:683] Acquiring locks for 3 operations
0905 19:59:42.254894 (+ 482us) tablet.cc:699] Row locks acquired
0905 19:59:42.254922 (+ 28us) write_op.cc:260] PREPARE: finished
0905 19:59:42.255119 (+ 197us) write_op.cc:270] Start()
0905 19:59:42.255270 (+ 151us) write_op.cc:276] Timestamp: P: 1757102382255055 usec, L: 0
0905 19:59:42.255299 (+ 29us) op_driver.cc:348] REPLICATION: starting
0905 19:59:42.318742 (+ 63443us) log.cc:844] Serialized 1317 byte log entry
0905 19:59:43.451139 (+1132397us) op_driver.cc:464] REPLICATION: finished
0905 19:59:44.035384 (+584245us) write_op.cc:301] APPLY: starting
0905 19:59:44.035478 (+ 94us) tablet.cc:1365] starting BulkCheckPresence
0905 19:59:44.035621 (+ 143us) tablet.cc:1368] finished BulkCheckPresence
0905 19:59:44.035721 (+ 100us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:59:44.039795 (+ 4074us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:59:44.039996 (+ 201us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0905 19:59:44.040042 (+ 46us) write_op.cc:312] APPLY: finished
0905 19:59:44.040257 (+ 215us) log.cc:844] Serialized 35 byte log entry
0905 19:59:44.040493 (+ 236us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:59:44.040601 (+ 108us) write_op.cc:454] Released schema lock
0905 19:59:44.040648 (+ 47us) write_op.cc:341] FINISH: Updating metrics
0905 19:59:44.043776 (+ 3128us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:59:44.051985 (+ 8209us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":18253,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":562567,"apply.run_cpu_time_us":2477,"apply.run_wall_time_us":5629,"num_ops":3,"prepare.queue_time_us":928695,"prepare.run_cpu_time_us":224356,"prepare.run_wall_time_us":613561,"raft.queue_time_us":555424,"raft.run_cpu_time_us":46113,"raft.run_wall_time_us":841976,"replication_time_us":1172810,"thread_start_us":1564442,"threads_started":5,"wal-append.queue_time_us":548208,"wal-append.run_cpu_time_us":39240,"wal-append.run_wall_time_us":3637664}]]}
I20250905 19:59:46.920291 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:46.852727 28078 raft_consensus.cc:2391] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 3: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:46.804338 30005 consensus_queue.cc:1046] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20250905 19:59:46.830947 30017 raft_consensus.cc:738] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:46.766279 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:46.880008 28001 rpcz_store.cc:276] 0905 19:59:45.723650 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:46.877972 (+1154322us) service_pool.cc:225] Handling call
0905 19:59:46.878339 (+ 367us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:46.879263 (+ 924us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:46.879479 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:46.933285 28001 raft_consensus.cc:1273] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250905 19:59:46.934852 28001 raft_consensus.cc:1273] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250905 19:59:46.814714 28122 leader_election.cc:304] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:46.881253 28154 raft_consensus.cc:2366] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 2. Current term is 3.
I20250905 19:59:46.745067 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 19:59:46.881951 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 71803.9us
I20250905 19:59:46.882038 28009 raft_consensus.cc:1238] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:46.833609 27908 rpcz_store.cc:276] 0905 19:59:39.026253 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:39.036965 (+ 10712us) service_pool.cc:225] Handling call
0905 19:59:46.760531 (+7723566us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:59:42.866383 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:59:43.240996 (+374613us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:59:43.243560 (+ 2564us) write_op.cc:435] Acquired schema lock
0905 19:59:43.243587 (+ 27us) tablet.cc:660] Decoding operations
0905 19:59:43.386907 (+143320us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:59:43.403374 (+ 16467us) write_op.cc:641] Partition lock acquired for write op
0905 19:59:43.420185 (+ 16811us) tablet.cc:683] Acquiring locks for 3 operations
0905 19:59:43.479640 (+ 59455us) tablet.cc:699] Row locks acquired
0905 19:59:43.479674 (+ 34us) write_op.cc:260] PREPARE: finished
0905 19:59:43.506344 (+ 26670us) write_op.cc:270] Start()
0905 19:59:43.521933 (+ 15589us) write_op.cc:276] Timestamp: P: 1757102383499971 usec, L: 0
0905 19:59:43.521979 (+ 46us) op_driver.cc:348] REPLICATION: starting
0905 19:59:44.042542 (+520563us) log.cc:844] Serialized 1317 byte log entry
0905 19:59:44.592795 (+550253us) spinlock_profiling.cc:243] Waited 337 ms on lock 0x7b54001bec54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc14de8 00007fe4a16434f1 00007fe4a1642cc8 00007fe4a1647961 00007fe4a1647919 00007fe4a16478a9 00007fe4a1647871 00007fe4a1646a9d 0000000000379254 0000000000379099
0905 19:59:44.846281 (+253486us) op_driver.cc:464] REPLICATION: finished
0905 19:59:45.435131 (+588850us) write_op.cc:301] APPLY: starting
0905 19:59:45.435230 (+ 99us) tablet.cc:1365] starting BulkCheckPresence
0905 19:59:45.435379 (+ 149us) tablet.cc:1368] finished BulkCheckPresence
0905 19:59:45.435405 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:59:45.435934 (+ 529us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:59:45.436130 (+ 196us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0905 19:59:45.436181 (+ 51us) write_op.cc:312] APPLY: finished
0905 19:59:45.436407 (+ 226us) log.cc:844] Serialized 35 byte log entry
0905 19:59:45.436602 (+ 195us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:59:45.436705 (+ 103us) write_op.cc:454] Released schema lock
0905 19:59:45.436754 (+ 49us) write_op.cc:341] FINISH: Updating metrics
0905 19:59:45.437133 (+ 379us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:59:45.442720 (+ 5587us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1645859,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":581284,"apply.run_cpu_time_us":1990,"apply.run_wall_time_us":1990,"num_ops":3,"prepare.queue_time_us":1599632,"prepare.run_cpu_time_us":16128,"prepare.run_wall_time_us":1760340,"replication_time_us":1313386,"spinlock_wait_cycles":774142208,"thread_start_us":325447,"threads_started":1}]]}
I20250905 19:59:46.937013 28009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1288) took 1213 ms. Trace:
I20250905 19:59:46.880015 28004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1290) took 1156 ms. Trace:
I20250905 19:59:46.883354 27907 catalog_manager.cc:5582] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 1 to 2, leader changed from e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 2 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20250905 19:59:46.937669 28004 rpcz_store.cc:276] 0905 19:59:45.724004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:46.878931 (+1154927us) service_pool.cc:225] Handling call
0905 19:59:46.879190 (+ 259us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:46.879788 (+ 598us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:46.879967 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:46.919627 30040 log.cc:927] Time spent T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
W20250905 19:59:46.891621 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:46.889902 28007 rpcz_store.cc:276] 0905 19:59:45.639168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:46.828862 (+1189694us) service_pool.cc:225] Handling call
0905 19:59:46.829304 (+ 442us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:46.889157 (+ 59853us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:46.889374 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:46.893101 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:46.946549 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:46.947979 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:46.906002 28006 raft_consensus.cc:1273] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250905 19:59:46.949395 28006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1285) took 1279 ms. Trace:
I20250905 19:59:46.949833 28006 rpcz_store.cc:276] 0905 19:59:45.669875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:46.872161 (+1202286us) service_pool.cc:225] Handling call
0905 19:59:46.901536 (+ 29375us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:46.949077 (+ 47541us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:46.949327 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:46.906469 28051 leader_election.cc:336] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 2.102s (SENT)
W20250905 19:59:46.951263 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 44873.1us
I20250905 19:59:46.910012 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:46.911161 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:46.912443 28749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40918 (request call id 454) took 2182 ms. Trace:
I20250905 19:59:46.955433 28749 rpcz_store.cc:276] 0905 19:59:44.722203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:44.722394 (+ 191us) service_pool.cc:225] Handling call
0905 19:59:44.749311 (+ 26917us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:45.147682 (+398371us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:45.147713 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:45.148159 (+ 446us) log.cc:844] Serialized 26 byte log entry
0905 19:59:45.350585 (+202426us) raft_consensus.cc:1643] Marking committed up to 4
0905 19:59:45.350785 (+ 200us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:46.133093 (+782308us) spinlock_profiling.cc:243] Waited 214 ms on lock 0x7b5400354b54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:46.133149 (+ 56us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:46.716881 (+583732us) spinlock_profiling.cc:243] Waited 6.04 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49d85891e 00007fe49d857c66 00007fe49d857888 00007fe49d8612c1 00007fe49d861279 00007fe49d861209
0905 19:59:46.903896 (+187015us) raft_consensus.cc:1683] finished
0905 19:59:46.903933 (+ 37us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:46.904143 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":331352,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":56264,"spinlock_wait_cycles":507117184,"thread_start_us":349966,"threads_started":2,"wal-append.queue_time_us":422692}
I20250905 19:59:46.856470 29917 raft_consensus.cc:738] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
W20250905 19:59:46.916239 27975 connection.cc:232] RPC call timeout handler was delayed by -3.9793928860000278s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1844s for a response
W20250905 19:59:46.958267 27975 connection.cc:232] RPC call timeout handler was delayed by -3.2566928859999962s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2647s for a response
I20250905 19:59:46.859145 30037 raft_consensus.cc:738] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:46.961584 28051 leader_election.cc:304] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:46.961278 30037 consensus_queue.cc:260] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:46.925379 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:46.925916 28002 raft_consensus.cc:1238] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:46.913444 28076 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Advancing to term 7
I20250905 19:59:46.926841 28008 raft_consensus.cc:1238] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:46.929803 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:46.907083 29906 raft_consensus.cc:2802] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election won for term 5
I20250905 19:59:46.938416 27907 catalog_manager.cc:5582] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 1 to 2, leader changed from e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 2 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:46.939250 28005 raft_consensus.cc:1238] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:46.903271 28003 raft_consensus.cc:1238] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:46.939508 28009 rpcz_store.cc:276] 0905 19:59:45.723329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:46.875310 (+1151981us) service_pool.cc:225] Handling call
0905 19:59:46.881816 (+ 6506us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:46.936647 (+ 54831us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:46.936928 (+ 281us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:46.901681 28745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:40926 (request call id 404) took 1254 ms. Trace:
I20250905 19:59:46.887272 29894 consensus_queue.cc:1046] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20250905 19:59:46.973855 28159 raft_consensus.cc:2391] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 3: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:47.024348 28007 raft_consensus.cc:1273] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250905 19:59:47.040218 28003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1283) took 1370 ms. Trace:
I20250905 19:59:47.041673 28005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1886) took 1066 ms. Trace:
I20250905 19:59:47.066365 28005 rpcz_store.cc:276] 0905 19:59:45.977755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:46.914992 (+937237us) service_pool.cc:225] Handling call
0905 19:59:46.939100 (+ 24108us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:47.041405 (+102305us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:47.041609 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:47.066299 28003 rpcz_store.cc:276] 0905 19:59:45.669283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:46.861353 (+1192070us) service_pool.cc:225] Handling call
0905 19:59:46.903026 (+ 41673us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:47.039892 (+136866us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:47.040147 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:47.048532 28157 raft_consensus.cc:2391] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 4: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
W20250905 19:59:47.049232 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 107906us
I20250905 19:59:47.068611 28159 raft_consensus.cc:1238] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:47.070456 28159 raft_consensus.cc:1238] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:47.071195 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:47.071470 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:47.072007 28158 raft_consensus.cc:2391] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 6: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:47.072536 28159 raft_consensus.cc:2391] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 5: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:46.967365 28156 raft_consensus.cc:2391] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 7: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:46.967006 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:47.079932 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:47.080588 28001 raft_consensus.cc:3058] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:47.081866 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:59:47.047137 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 88044.7us
I20250905 19:59:47.084043 28009 raft_consensus.cc:1273] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:59:47.077479 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:47.087805 28007 raft_consensus.cc:2391] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:47.080782 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:47.088660 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:47.089025 28008 raft_consensus.cc:3058] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 3
I20250905 19:59:47.089723 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:47.091368 28152 raft_consensus.cc:1238] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 4. Current term is 5. Ops: []
I20250905 19:59:47.107192 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:46.930567 29987 raft_consensus.cc:695] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:47.118813 28001 raft_consensus.cc:2466] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 19:59:47.120142 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:47.120711 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:47.120882 28001 raft_consensus.cc:3058] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:47.021049 29824 consensus_queue.cc:1046] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.101s
I20250905 19:59:47.122704 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:47.126097 28003 raft_consensus.cc:2391] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 6: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 19:59:47.126063 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:47.128362 28157 raft_consensus.cc:2391] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 4: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:47.130796 28154 raft_consensus.cc:1238] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:47.148240 28125 leader_election.cc:304] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:47.154218 28718 leader_election.cc:304] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:47.159999 28005 raft_consensus.cc:1273] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250905 19:59:47.163517 29871 raft_consensus.cc:491] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:47.164295 29871 raft_consensus.cc:513] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:47.106369 29894 consensus_queue.cc:1230] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer 26b2f874b8e54f08b01a9ed00d5cce05 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20250905 19:59:47.108247 29934 leader_election.cc:290] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:46.922605 29864 consensus_queue.cc:260] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:47.195191 28003 raft_consensus.cc:1273] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250905 19:59:47.197779 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:47.198861 28009 raft_consensus.cc:2391] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:47.200282 28009 raft_consensus.cc:1273] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250905 19:59:47.202751 28008 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:47.204346 28008 raft_consensus.cc:1238] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:47.162231 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:47.042227 28745 rpcz_store.cc:276] 0905 19:59:45.613059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.613190 (+ 131us) service_pool.cc:225] Handling call
0905 19:59:46.620826 (+1007636us) spinlock_profiling.cc:243] Waited 269 ms on lock 0x7b54004b00d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc109fd 00007fe4a2b7f890 00007fe49c210c44 00007fe49c210bd2 00007fe49c210b01 00007fe49c210a7c 00007fe49c20fd32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:46.858917 (+238091us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":618312448}
I20250905 19:59:47.206087 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:47.263449 28005 raft_consensus.cc:3058] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:46.940966 29809 consensus_queue.cc:260] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:47.196806 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:47.201661 28009 raft_consensus.cc:1238] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:47.229167 28718 leader_election.cc:304] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:47.271236 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 19:59:47.230799 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 416120us
I20250905 19:59:47.256058 29756 raft_consensus.cc:3053] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:59:47.273172 29756 raft_consensus.cc:738] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:47.166446 29840 consensus_queue.cc:1046] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.094s
I20250905 19:59:47.274400 29894 raft_consensus.cc:2802] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:47.274025 29756 consensus_queue.cc:260] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:47.276525 28080 raft_consensus.cc:1238] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 2. Current term is 3. Ops: []
W20250905 19:59:47.207753 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 290969us
I20250905 19:59:47.268477 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:47.284511 28157 raft_consensus.cc:2391] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 4: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 19:59:47.166678 30036 raft_consensus.cc:2747] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 19:59:47.177696 28001 raft_consensus.cc:2466] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 19:59:47.193882 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:47.293900 28154 raft_consensus.cc:3053] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Stepping down as leader of term 2
W20250905 19:59:47.294052 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:47.294472 28154 raft_consensus.cc:738] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:47.295399 28154 consensus_queue.cc:260] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:47.296319 28154 raft_consensus.cc:3058] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 4
I20250905 19:59:47.182299 29848 raft_consensus.cc:695] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:47.240064 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:47.285202 28122 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:47.258229 29934 raft_consensus.cc:3053] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:47.301333 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:47.258877 29864 raft_consensus.cc:3058] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:47.302563 28159 raft_consensus.cc:2391] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 7: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:47.159489 29917 consensus_queue.cc:260] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:47.275221 29756 raft_consensus.cc:3058] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:47.326869 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:47.327764 28159 raft_consensus.cc:2391] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 3: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:47.331314 28005 raft_consensus.cc:2466] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 19:59:47.301913 29934 raft_consensus.cc:738] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
W20250905 19:59:47.335307 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 50188.8us
W20250905 19:59:47.317504 30047 log.cc:927] Time spent T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.440s user 0.001s sys 0.000s
I20250905 19:59:47.336478 28122 leader_election.cc:304] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:47.342371 28154 raft_consensus.cc:2466] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 19:59:47.363695 28122 leader_election.cc:304] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:47.310345 30005 consensus_queue.cc:1035] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.048s
W20250905 19:59:47.350037 30050 log.cc:927] Time spent T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.167s user 0.000s sys 0.000s
I20250905 19:59:47.376036 28122 leader_election.cc:304] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:47.365408 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 19:59:47.368275 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 215643us
I20250905 19:59:47.332554 30059 raft_consensus.cc:2802] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:47.379582 28749 raft_consensus.cc:1238] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 LEADER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20250905 19:59:47.364720 29906 raft_consensus.cc:695] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
W20250905 19:59:47.419976 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:47.343122 30061 consensus_queue.cc:1046] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20250905 19:59:47.366300 30057 raft_consensus.cc:2802] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader election won for term 7
I20250905 19:59:47.382218 30058 raft_consensus.cc:3058] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
W20250905 19:59:47.430382 28153 consensus_meta.cc:287] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.549s user 0.014s sys 0.002s
I20250905 19:59:47.315764 29987 consensus_queue.cc:237] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:47.431789 28153 raft_consensus.cc:2466] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 19:59:47.385027 28745 raft_consensus.cc:1238] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 2 LEADER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 1. Current term is 2. Ops: []
W20250905 19:59:47.448444 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:47.432559 30052 log.cc:927] Time spent T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.115s user 0.000s sys 0.001s
I20250905 19:59:47.451591 30067 raft_consensus.cc:2802] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:47.451941 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 19:59:47.453235 30055 log.cc:927] Time spent T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.142s user 0.000s sys 0.001s
I20250905 19:59:47.379572 29943 raft_consensus.cc:491] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:47.456741 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 19:59:47.458276 30056 log.cc:927] Time spent T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20250905 19:59:47.459128 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:47.421291 29934 consensus_queue.cc:260] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:47.493960 29934 raft_consensus.cc:3058] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:47.494086 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:47.494958 28157 raft_consensus.cc:2391] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 4: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:47.459498 28002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 199) took 1815 ms. Trace:
I20250905 19:59:47.496527 28002 rpcz_store.cc:276] 0905 19:59:45.644071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:46.831531 (+1187460us) service_pool.cc:225] Handling call
0905 19:59:46.925745 (+ 94214us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:47.078094 (+152349us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:47.459289 (+381195us) spinlock_profiling.cc:243] Waited 68.8 ms on lock 0x7b54001fbb50. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:47.459438 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":158131584}
I20250905 19:59:47.461090 30017 consensus_queue.cc:260] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:47.466836 29854 raft_consensus.cc:2802] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:47.498395 29854 raft_consensus.cc:695] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:47.499615 29854 consensus_queue.cc:237] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:47.468149 29900 consensus_meta.cc:287] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.635s user 0.011s sys 0.003s
I20250905 19:59:47.468379 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:47.469703 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:47.504787 28156 raft_consensus.cc:1273] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250905 19:59:47.472131 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:47.389999 29871 leader_election.cc:290] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:47.480276 28151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 1944) took 1413 ms. Trace:
I20250905 19:59:47.506842 29871 consensus_queue.cc:1046] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
W20250905 19:59:47.506273 27973 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:47.367660 29848 consensus_queue.cc:237] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:47.508625 27973 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:47.512156 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:47.515594 28080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1356) took 4050 ms. Trace:
I20250905 19:59:47.516183 28080 rpcz_store.cc:276] 0905 19:59:43.465388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.654595 (+2189207us) service_pool.cc:225] Handling call
0905 19:59:46.210513 (+555918us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:47.277033 (+1066520us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:47.349582 (+ 72549us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54003aa054. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:47.515299 (+165717us) spinlock_profiling.cc:243] Waited 537 ms on lock 0x7b54003aa050. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:47.515491 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3686259968}
W20250905 19:59:47.490300 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 109975us
I20250905 19:59:47.506731 28151 rpcz_store.cc:276] 0905 19:59:46.066941 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:46.295070 (+228129us) service_pool.cc:225] Handling call
0905 19:59:46.302970 (+ 7900us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:46.872469 (+569499us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:47.479914 (+607445us) spinlock_profiling.cc:243] Waited 496 ms on lock 0x7b54000ed8d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:47.480186 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1141562240}
W20250905 19:59:47.436013 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 164057us
I20250905 19:59:47.521236 28718 leader_election.cc:304] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:47.525272 28718 leader_election.cc:304] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:47.526881 28718 leader_election.cc:304] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:47.518512 30067 raft_consensus.cc:695] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:47.538837 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:47.546599 28002 raft_consensus.cc:3053] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:47.547108 28002 raft_consensus.cc:738] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
W20250905 19:59:47.536476 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 148251us
I20250905 19:59:47.519629 30063 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:47.549575 28749 raft_consensus.cc:2391] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 2: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 19:59:47.532032 29936 raft_consensus.cc:2747] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 19:59:47.532900 29884 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:47.544373 29894 raft_consensus.cc:695] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:47.538443 29840 raft_consensus.cc:2802] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:47.577502 28081 raft_consensus.cc:1238] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:47.535960 30070 raft_consensus.cc:2802] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:47.517867 29943 raft_consensus.cc:513] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:47.558312 28002 consensus_queue.cc:260] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:47.527961 30069 consensus_queue.cc:1046] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
W20250905 19:59:47.585633 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 207992us
I20250905 19:59:47.595917 28122 leader_election.cc:304] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:47.597829 28122 leader_election.cc:304] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
W20250905 19:59:47.586867 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 625350us
I20250905 19:59:47.599867 28051 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 19:59:47.587510 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 539528us
W20250905 19:59:47.591217 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:47.588636 29931 raft_consensus.cc:2802] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 19:59:47.537988 30036 consensus_queue.cc:1046] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.040s
I20250905 19:59:47.583376 28002 raft_consensus.cc:3058] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:59:47.586453 29848 raft_consensus.cc:3053] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:47.606274 29848 raft_consensus.cc:738] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
W20250905 19:59:47.593124 29964 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.725s user 0.012s sys 0.014s
I20250905 19:59:47.553366 30063 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:47.605005 29931 raft_consensus.cc:491] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:47.623046 29931 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 19:59:47.624068 30000 raft_consensus.cc:3053] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 4 LEADER]: Stepping down as leader of term 4
I20250905 19:59:47.624485 30000 raft_consensus.cc:738] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:47.625314 30000 consensus_queue.cc:260] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:47.627202 28743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40904 (request call id 501) took 2517 ms. Trace:
W20250905 19:59:47.611414 29878 consensus_meta.cc:287] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.866s user 0.012s sys 0.000s
W20250905 19:59:47.621974 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 88455.8us
W20250905 19:59:47.617259 29964 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.755s user 0.012s sys 0.014s
I20250905 19:59:47.614914 30069 consensus_queue.cc:1230] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Peer 7e9c9f591f05464f9b097bf003d14f4a log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250905 19:59:47.602700 28744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40926 (request call id 407) took 1593 ms. Trace:
W20250905 19:59:47.688481 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 47037us
I20250905 19:59:47.629325 30061 consensus_queue.cc:1230] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer cd764bfeebac4cc9ad851cd4a6a7d3b0 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250905 19:59:47.578655 29884 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:47.574625 29906 consensus_queue.cc:237] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:47.607812 29848 consensus_queue.cc:260] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 19:59:47.627694 28743 rpcz_store.cc:276] 0905 19:59:45.109832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.118469 (+ 8637us) service_pool.cc:225] Handling call
0905 19:59:46.020289 (+901820us) spinlock_profiling.cc:243] Waited 7.93 ms on lock 0x7b5800304598. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b91f60 00007fe4a2b7d1a9 00007fe4a2b7cc52 00007fe49c202b44
0905 19:59:46.020541 (+ 252us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:46.846674 (+826133us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:47.626846 (+780172us) spinlock_profiling.cc:243] Waited 722 ms on lock 0x7b5400570dd4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:47.627100 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1677876224}
I20250905 19:59:47.651822 30000 raft_consensus.cc:3058] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:47.632622 30067 consensus_queue.cc:237] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:47.717257 29840 raft_consensus.cc:695] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:47.723255 29840 consensus_queue.cc:237] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:47.650746 29878 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:47.711390 30061 raft_consensus.cc:2747] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 19:59:47.726861 30074 raft_consensus.cc:2747] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:47.741657 30089 raft_consensus.cc:3053] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Stepping down as leader of term 4
I20250905 19:59:47.742843 30086 consensus_queue.cc:1035] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20250905 19:59:47.745914 29943 leader_election.cc:290] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:47.747431 30063 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:47.748472 30017 raft_consensus.cc:3058] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
W20250905 19:59:47.751996 30032 consensus_meta.cc:287] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.824s user 0.011s sys 0.004s
I20250905 19:59:47.732578 30070 raft_consensus.cc:695] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:47.732461 30080 consensus_queue.cc:1035] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20250905 19:59:47.741020 30079 consensus_queue.cc:1035] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
W20250905 19:59:47.299327 30053 consensus_queue.cc:1162] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7e9c9f591f05464f9b097bf003d14f4a" responder_term: 4 status { last_received { term: 2 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250905 19:59:47.721685 30036 raft_consensus.cc:3053] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:59:47.756853 30089 raft_consensus.cc:738] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:47.773802 30070 consensus_queue.cc:237] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:47.779755 30089 consensus_queue.cc:260] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:47.783156 30089 raft_consensus.cc:3058] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Advancing to term 5
W20250905 19:59:47.784770 30062 log.cc:927] Time spent T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.425s user 0.003s sys 0.000s
W20250905 19:59:47.816442 27973 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:47.819449 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:47.819932 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:47.820271 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:47.832379 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 228855us
I20250905 19:59:47.835886 28083 raft_consensus.cc:2391] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 5: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:47.836459 28122 leader_election.cc:304] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
W20250905 19:59:47.839815 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:47.846786 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 145764us
I20250905 19:59:47.849875 28125 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:47.852674 28081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1359) took 4385 ms. Trace:
I20250905 19:59:47.870708 28081 rpcz_store.cc:276] 0905 19:59:43.466846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.677672 (+2210826us) service_pool.cc:225] Handling call
0905 19:59:46.572599 (+894927us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:47.578158 (+1005559us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:47.851262 (+273104us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b540019ebd4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:47.852223 (+ 961us) spinlock_profiling.cc:243] Waited 895 ms on lock 0x7b540019ebd0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:47.852451 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4367750912}
I20250905 19:59:47.852659 28010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1287) took 2129 ms. Trace:
I20250905 19:59:47.865082 28083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1363) took 1125 ms. Trace:
I20250905 19:59:47.873204 28083 rpcz_store.cc:276] 0905 19:59:46.739762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:46.807804 (+ 68042us) service_pool.cc:225] Handling call
0905 19:59:47.864597 (+1056793us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b540029a254. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:47.864928 (+ 331us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2359572736}
I20250905 19:59:47.876807 28150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 1955) took 1326 ms. Trace:
W20250905 19:59:47.880885 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:47.881114 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:47.882745 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:47.883248 28747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40904 (request call id 506) took 1897 ms. Trace:
I20250905 19:59:47.883860 28747 rpcz_store.cc:276] 0905 19:59:45.985772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:45.987844 (+ 2072us) service_pool.cc:225] Handling call
0905 19:59:46.734177 (+746333us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:46.825680 (+ 91503us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:47.881634 (+1055954us) spinlock_profiling.cc:243] Waited 725 ms on lock 0x7b540069c8d0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:47.882976 (+ 1342us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1668116352}
W20250905 19:59:47.884955 27973 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:47.696437 28744 rpcz_store.cc:276] 0905 19:59:46.009325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:46.016457 (+ 7132us) service_pool.cc:225] Handling call
0905 19:59:46.016852 (+ 395us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:46.773428 (+756576us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:47.579249 (+805821us) spinlock_profiling.cc:243] Waited 697 ms on lock 0x7b5400570154. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:47.599660 (+ 20411us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1603792256}
I20250905 19:59:47.887138 29943 raft_consensus.cc:491] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:47.734243 29848 raft_consensus.cc:3058] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:47.749277 29871 raft_consensus.cc:3053] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:59:47.895109 29864 consensus_queue.cc:1035] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:47.895944 29871 raft_consensus.cc:738] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:47.699951 29894 consensus_queue.cc:237] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:47.899793 29809 raft_consensus.cc:3058] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:47.746122 29884 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:47.901454 29884 consensus_queue.cc:1046] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250905 19:59:47.902092 29955 raft_consensus.cc:491] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:47.903379 29884 raft_consensus.cc:3053] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:59:47.734644 29813 raft_consensus.cc:491] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:47.903993 29992 raft_consensus.cc:3053] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:59:47.904724 29992 raft_consensus.cc:738] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:47.904973 29813 raft_consensus.cc:513] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:47.747073 29917 raft_consensus.cc:3058] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
W20250905 19:59:47.583907 29924 consensus_queue.cc:1162] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7e9c9f591f05464f9b097bf003d14f4a" responder_term: 2 status { last_received { term: 2 index: 5 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250905 19:59:47.891232 29853 consensus_queue.cc:1035] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 19:59:47.766782 30032 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:47.911873 29644 raft_consensus.cc:3058] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:47.916690 30032 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:47.916380 29992 consensus_queue.cc:260] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:47.897974 29871 consensus_queue.cc:260] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:47.922756 29871 raft_consensus.cc:3058] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:47.923821 30036 raft_consensus.cc:738] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:47.925035 30037 raft_consensus.cc:3058] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:47.918035 29992 raft_consensus.cc:3058] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Advancing to term 4
W20250905 19:59:47.936040 27973 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:47.888307 29943 raft_consensus.cc:513] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:47.903998 29884 raft_consensus.cc:738] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:47.938925 29955 raft_consensus.cc:513] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:48.000326 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 60625.7us
W20250905 19:59:48.002197 29905 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.207s user 0.001s sys 0.012s
I20250905 19:59:48.004436 30094 raft_consensus.cc:491] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:48.005084 30054 raft_consensus.cc:2802] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:48.036291 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:48.037211 28743 raft_consensus.cc:2466] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
W20250905 19:59:48.005559 30059 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.521s user 0.009s sys 0.009s
W20250905 19:59:48.037962 30059 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.554s user 0.009s sys 0.009s
W20250905 19:59:48.006938 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 387498us
I20250905 19:59:48.007196 28748 raft_consensus.cc:1238] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 6. Current term is 7. Ops: []
W20250905 19:59:48.007623 30058 consensus_meta.cc:287] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.569s user 0.010s sys 0.002s
I20250905 19:59:48.010596 29964 consensus_queue.cc:1046] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.021s
I20250905 19:59:48.011365 28006 raft_consensus.cc:2953] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:48.015578 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:48.018862 30092 raft_consensus.cc:491] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:47.872311 28010 rpcz_store.cc:276] 0905 19:59:45.722930 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:46.874409 (+1151479us) service_pool.cc:225] Handling call
0905 19:59:46.874711 (+ 302us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:47.515407 (+640696us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:47.515455 (+ 48us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:47.515517 (+ 62us) raft_consensus.cc:1643] Marking committed up to 1
0905 19:59:47.515582 (+ 65us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:47.852102 (+336520us) spinlock_profiling.cc:243] Waited 630 ms on lock 0x7b54001fc054. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:47.852177 (+ 75us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:47.852440 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10388,"raft.queue_time_us":19988,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":5678,"spinlock_wait_cycles":1448648448}
I20250905 19:59:48.022658 29878 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 19:59:48.025236 28076 consensus_meta.cc:287] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.939s user 0.016s sys 0.011s
I20250905 19:59:48.026172 30080 raft_consensus.cc:491] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:48.027213 30070 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:48.027470 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 84776.7us
I20250905 19:59:48.028672 30085 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:48.029453 29934 consensus_queue.cc:1035] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250905 19:59:48.030560 29933 raft_consensus.cc:2747] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:48.037442 30054 raft_consensus.cc:491] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:48.031589 30091 raft_consensus.cc:491] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:48.031949 30005 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:48.033926 29905 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.267s user 0.002s sys 0.012s
I20250905 19:59:48.058383 30005 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:48.031597 30093 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:48.059969 30093 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:48.063483 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:48.064221 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:48.064427 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:48.065102 28744 raft_consensus.cc:2466] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
I20250905 19:59:48.066565 28751 raft_consensus.cc:1238] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:48.067832 28122 leader_election.cc:304] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:48.038331 30059 raft_consensus.cc:695] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:48.057029 28076 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 7.
I20250905 19:59:48.071146 30059 consensus_queue.cc:237] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:48.071990 29943 leader_election.cc:290] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:47.926226 30036 consensus_queue.cc:260] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:48.072803 28076 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50660 (request call id 192) took 1305 ms. Trace:
I20250905 19:59:48.073621 30036 raft_consensus.cc:3058] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:48.074661 28749 raft_consensus.cc:3058] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:48.077623 30072 raft_consensus.cc:491] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:48.078446 30072 raft_consensus.cc:513] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:48.073290 28076 rpcz_store.cc:276] 0905 19:59:46.768346 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:46.775127 (+ 6781us) service_pool.cc:225] Handling call
0905 19:59:48.072706 (+1297579us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:47.878880 28150 rpcz_store.cc:276] 0905 19:59:46.550823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:46.723229 (+172406us) service_pool.cc:225] Handling call
0905 19:59:46.723510 (+ 281us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:46.723738 (+ 228us) raft_consensus.cc:1532] Early marking committed up to index 2
0905 19:59:46.844258 (+120520us) log.cc:844] Serialized 14 byte log entry
0905 19:59:47.875514 (+1031256us) spinlock_profiling.cc:243] Waited 7.52 ms on lock 0x7b180032dca0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49d86555c 00007fe49ac873cb 00007fe4a38dcb99 00007fe49aca369b 00007fe49aca193f 00007fe49aca1455 00007fe49dc11020 00007fe49dc236c6 00007fe49dc34237
0905 19:59:47.875787 (+ 273us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:47.875934 (+ 147us) raft_consensus.cc:1643] Marking committed up to 2
0905 19:59:47.876070 (+ 136us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:47.876243 (+ 173us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:47.876498 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":297553,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":12954,"raft.queue_time_us":113782,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":75366,"spinlock_wait_cycles":17297664,"thread_start_us":378115,"threads_started":2}
I20250905 19:59:48.082333 29878 raft_consensus.cc:491] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:48.052340 30032 consensus_queue.cc:1035] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.031s
I20250905 19:59:48.082369 29995 raft_consensus.cc:491] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:48.084550 29995 raft_consensus.cc:513] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 19:59:48.085407 29884 consensus_queue.cc:260] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:48.086573 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:48.087003 29884 raft_consensus.cc:3058] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 1 FOLLOWER]: Advancing to term 2
W20250905 19:59:48.087256 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:48.047772 27974 leader_election.cc:304] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:48.072033 29941 raft_consensus.cc:2747] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250905 19:59:48.083143 29878 raft_consensus.cc:513] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:48.034636 30094 raft_consensus.cc:513] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:48.034907 29813 leader_election.cc:290] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:48.064455 28005 raft_consensus.cc:2466] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 2.
I20250905 19:59:48.032279 30053 raft_consensus.cc:2747] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250905 19:59:48.096482 28747 raft_consensus.cc:1238] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:48.096218 30075 consensus_queue.cc:1046] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20250905 19:59:48.096129 30070 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:48.055729 30054 raft_consensus.cc:3058] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:48.098156 27975 leader_election.cc:304] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
W20250905 19:59:48.098389 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 360525us
I20250905 19:59:48.098649 29864 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:48.098865 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 398612us
I20250905 19:59:48.099092 30091 raft_consensus.cc:513] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:48.102046 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:48.097446 30080 raft_consensus.cc:513] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:48.113095 30070 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:48.114316 28051 leader_election.cc:400] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Vote denied by peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421) with higher term. Message: Invalid argument: T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4.
I20250905 19:59:48.115015 28749 raft_consensus.cc:2466] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 19:59:48.116818 30094 leader_election.cc:290] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:48.117252 29878 leader_election.cc:290] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:48.117033 29864 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:48.150035 29853 raft_consensus.cc:491] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:48.150480 29864 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:48.117383 30099 consensus_queue.cc:1046] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20250905 19:59:48.151031 29853 raft_consensus.cc:513] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:48.151798 30099 consensus_queue.cc:1230] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer 7e9c9f591f05464f9b097bf003d14f4a log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250905 19:59:48.117764 30079 raft_consensus.cc:3053] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:48.153194 30079 raft_consensus.cc:738] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:48.153332 29853 leader_election.cc:290] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:48.154253 30079 consensus_queue.cc:260] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:48.138373 28051 leader_election.cc:304] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:48.155747 30079 raft_consensus.cc:3058] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:48.118183 30092 raft_consensus.cc:513] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:48.119053 29931 raft_consensus.cc:513] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:48.123404 30081 raft_consensus.cc:3053] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Stepping down as leader of term 2
W20250905 19:59:48.124634 30057 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.584s user 0.005s sys 0.012s
I20250905 19:59:48.056684 30058 raft_consensus.cc:2747] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250905 19:59:48.172703 30057 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.718s user 0.005s sys 0.012s
I20250905 19:59:48.092697 30005 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:48.129458 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 19:59:48.173130 30057 raft_consensus.cc:695] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:48.174628 30057 consensus_queue.cc:237] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:48.129558 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:48.131718 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:48.132504 30080 leader_election.cc:290] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:48.136721 29906 raft_consensus.cc:2802] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:48.137085 29995 leader_election.cc:290] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421)
I20250905 19:59:48.137338 29943 raft_consensus.cc:2747] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250905 19:59:48.138362 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:48.137105 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 153263us
I20250905 19:59:48.139046 28743 raft_consensus.cc:1238] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 1. Current term is 2. Ops: []
I20250905 19:59:48.135728 30093 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:48.142803 30077 raft_consensus.cc:491] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:48.143908 28001 raft_consensus.cc:1238] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:48.156769 28051 leader_election.cc:304] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:48.138538 28123 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
W20250905 19:59:48.159260 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:48.168177 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 100434us
I20250905 19:59:48.170974 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:48.171670 30092 leader_election.cc:290] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:48.171919 30067 raft_consensus.cc:3058] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:48.172127 30081 raft_consensus.cc:738] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
W20250905 19:59:48.128089 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 85710.1us
I20250905 19:59:48.180505 28010 raft_consensus.cc:2466] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 19:59:48.180931 28718 leader_election.cc:304] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:48.183053 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:48.183947 30058 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:48.183017 30072 leader_election.cc:290] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:48.184756 29964 raft_consensus.cc:491] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:48.185385 30108 raft_consensus.cc:491] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:48.186079 30005 raft_consensus.cc:491] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:48.187093 29931 leader_election.cc:290] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:48.257208 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:48.258234 30058 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:48.259649 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 79072.5us
I20250905 19:59:48.189072 28150 raft_consensus.cc:1273] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250905 19:59:48.260979 30058 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:48.261333 30093 raft_consensus.cc:2802] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:48.196065 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:48.261811 30093 raft_consensus.cc:491] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:48.198302 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:48.203977 28158 raft_consensus.cc:1273] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20250905 19:59:48.190140 29941 raft_consensus.cc:2747] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:48.262928 28158 pending_rounds.cc:85] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting all ops after (but not including) 2
I20250905 19:59:48.133491 29955 leader_election.cc:290] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:48.263283 28158 pending_rounds.cc:107] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20250905 19:59:48.263651 28005 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 19:59:48.264644 30080 raft_consensus.cc:2802] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election won for term 5
I20250905 19:59:48.264921 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:48.265292 30080 raft_consensus.cc:695] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:48.265805 28005 raft_consensus.cc:2466] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 19:59:48.235458 29906 raft_consensus.cc:491] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250905 19:59:48.253566 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 118512us
I20250905 19:59:48.256477 28155 raft_consensus.cc:1238] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250905 19:59:48.262270 30093 raft_consensus.cc:3058] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:48.271587 30077 raft_consensus.cc:513] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:48.272102 30108 raft_consensus.cc:513] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:48.273728 29906 raft_consensus.cc:3058] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:48.269863 28122 leader_election.cc:304] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:48.267228 27974 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:48.262533 30081 consensus_queue.cc:260] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:48.209328 30096 log.cc:927] Time spent T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410: Append to log took a long time: real 0.084s user 0.000s sys 0.001s
W20250905 19:59:48.267606 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:48.263681 28158 raft_consensus.cc:2887] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250905 19:59:48.257091 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:48.269805 29964 raft_consensus.cc:513] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:48.257328 30005 raft_consensus.cc:513] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:48.278209 30102 raft_consensus.cc:2747] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250905 19:59:48.278666 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 122079us
I20250905 19:59:48.233446 30085 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:48.278708 29943 raft_consensus.cc:2802] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 19:59:48.279393 29931 raft_consensus.cc:491] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:48.282071 27973 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:48.297211 27973 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:48.286793 29871 raft_consensus.cc:3058] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:48.247663 29995 raft_consensus.cc:491] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:48.294468 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:48.295682 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:48.300948 28150 raft_consensus.cc:1273] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250905 19:59:48.302057 30077 leader_election.cc:290] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:48.302692 29853 consensus_queue.cc:1046] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 19:59:48.307389 28083 raft_consensus.cc:1238] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20250905 19:59:48.307493 28010 raft_consensus.cc:1273] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250905 19:59:48.327414 29964 leader_election.cc:290] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:48.308495 28081 raft_consensus.cc:1238] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 6. Current term is 7. Ops: []
I20250905 19:59:48.308633 29928 raft_consensus.cc:491] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:48.309521 29905 consensus_queue.cc:1046] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250905 19:59:48.309803 30091 leader_election.cc:290] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:48.310024 28051 leader_election.cc:304] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:48.310587 29955 consensus_queue.cc:1035] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:48.311367 29864 raft_consensus.cc:491] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:48.312109 28151 raft_consensus.cc:2466] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 19:59:48.276531 30080 consensus_queue.cc:237] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:48.315151 30005 leader_election.cc:290] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:48.324079 28157 raft_consensus.cc:2466] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 2.
I20250905 19:59:48.296756 29943 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:48.353374 29943 raft_consensus.cc:3058] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Advancing to term 7
I20250905 19:59:48.307680 30054 raft_consensus.cc:513] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:48.349620 28125 leader_election.cc:304] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:48.342051 29931 raft_consensus.cc:513] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:48.350585 29995 raft_consensus.cc:513] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:48.351699 30017 consensus_meta.cc:287] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.586s user 0.011s sys 0.013s
I20250905 19:59:48.351418 30091 consensus_queue.cc:1035] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250905 19:59:48.351789 29905 raft_consensus.cc:3053] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 LEADER]: Stepping down as leader of term 6
W20250905 19:59:48.352214 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:48.332074 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:48.308688 30085 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:48.342525 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:48.365546 28158 raft_consensus.cc:2466] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
W20250905 19:59:48.365929 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:48.367431 28155 raft_consensus.cc:2391] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 5: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:48.338699 30102 raft_consensus.cc:3053] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:59:48.372068 29853 raft_consensus.cc:2802] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:48.372526 30106 consensus_queue.cc:1046] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
W20250905 19:59:48.372731 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 106016us
I20250905 19:59:48.374536 28004 raft_consensus.cc:1273] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
W20250905 19:59:48.373397 30089 consensus_meta.cc:287] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.589s user 0.008s sys 0.007s
W20250905 19:59:48.377544 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:48.379458 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:48.401510 28001 raft_consensus.cc:2466] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
W20250905 19:59:48.406801 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:48.400583 28004 pending_rounds.cc:85] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Aborting all ops after (but not including) 4
I20250905 19:59:48.411758 28004 pending_rounds.cc:107] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20250905 19:59:48.412160 28004 raft_consensus.cc:2887] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250905 19:59:48.368818 28076 raft_consensus.cc:1238] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20250905 19:59:48.369098 29905 raft_consensus.cc:738] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:48.377110 30058 raft_consensus.cc:2747] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250905 19:59:48.419319 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:48.425649 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 115784us
I20250905 19:59:48.376789 30102 raft_consensus.cc:738] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:48.375044 29931 leader_election.cc:290] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 19:59:48.416957 29644 consensus_meta.cc:287] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.502s user 0.017s sys 0.000s
I20250905 19:59:48.363454 30077 consensus_queue.cc:1046] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
I20250905 19:59:48.435611 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:48.421603 27975 leader_election.cc:304] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:48.363372 29864 raft_consensus.cc:513] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:48.443542 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 126696us
I20250905 19:59:48.362018 29928 raft_consensus.cc:513] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:48.444921 28125 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:48.446616 28717 leader_election.cc:304] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 19:59:48.448072 28718 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:48.403580 30108 leader_election.cc:290] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 19:59:48.452230 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:48.452232 28717 leader_election.cc:304] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
W20250905 19:59:48.449430 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 50335.2us
I20250905 19:59:48.423534 29853 raft_consensus.cc:695] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:48.448297 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:48.457551 29848 consensus_queue.cc:1035] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.077s
I20250905 19:59:48.454092 30089 raft_consensus.cc:2802] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:48.451635 30037 consensus_queue.cc:1035] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.013s
I20250905 19:59:48.470688 28743 raft_consensus.cc:1273] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20250905 19:59:48.471378 28743 pending_rounds.cc:85] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a: Aborting all ops after (but not including) 5
W20250905 19:59:48.471482 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 101701us
I20250905 19:59:48.471657 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:48.472506 28158 raft_consensus.cc:2466] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 1.
I20250905 19:59:48.472666 29964 raft_consensus.cc:3053] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:48.473173 27975 leader_election.cc:400] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705) with higher term. Message: Invalid argument: T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6.
I20250905 19:59:48.458938 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 19:59:48.474591 28002 consensus_meta.cc:287] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.869s user 0.023s sys 0.003s
I20250905 19:59:48.474761 29924 raft_consensus.cc:2747] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 19:59:48.463482 29905 consensus_queue.cc:260] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:48.475363 28002 raft_consensus.cc:1273] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250905 19:59:48.475852 29905 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Advancing to term 7
I20250905 19:59:48.477068 28002 raft_consensus.cc:1273] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250905 19:59:48.463927 30089 raft_consensus.cc:491] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:48.464095 30091 raft_consensus.cc:2802] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 19:59:48.479506 30091 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:48.467694 29853 consensus_queue.cc:237] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:48.458352 30102 consensus_queue.cc:260] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:48.463572 30112 log.cc:927] Time spent T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.167s user 0.001s sys 0.000s
I20250905 19:59:48.458354 30004 raft_consensus.cc:491] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:48.481055 30102 raft_consensus.cc:3058] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:48.481187 30004 raft_consensus.cc:513] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:48.483127 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:48.484280 30004 leader_election.cc:290] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:48.484467 28003 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 19:59:48.479096 30089 raft_consensus.cc:3058] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:48.473140 29964 raft_consensus.cc:738] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:48.458716 30017 raft_consensus.cc:491] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:48.480065 30091 raft_consensus.cc:3058] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Advancing to term 5
W20250905 19:59:48.487367 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:48.490296 27975 leader_election.cc:400] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705) with higher term. Message: Invalid argument: T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 2. Current term is 3.
I20250905 19:59:48.533802 27975 leader_election.cc:403] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250905 19:59:48.458566 28084 raft_consensus.cc:1273] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20250905 19:59:48.494139 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 49291.8us
I20250905 19:59:48.450706 30000 raft_consensus.cc:491] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:48.493642 28718 leader_election.cc:304] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:48.541299 28001 raft_consensus.cc:2466] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 19:59:48.471679 28743 pending_rounds.cc:107] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a: Aborting uncommitted NO_OP operation due to leader change: 4.6
W20250905 19:59:48.536386 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:48.546931 29924 raft_consensus.cc:491] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:48.547667 28748 raft_consensus.cc:1215] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Deduplicated request from leader. Original: 1.4->[3.5-3.5] Dedup: 3.5->[]
I20250905 19:59:48.546949 30037 raft_consensus.cc:491] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:48.570299 29928 leader_election.cc:290] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:48.570405 30037 raft_consensus.cc:513] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:48.573746 30037 leader_election.cc:290] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:48.574590 30072 raft_consensus.cc:2802] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 19:59:48.544274 28007 raft_consensus.cc:2466] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 19:59:48.576031 30058 raft_consensus.cc:491] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:48.576922 30058 raft_consensus.cc:513] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:48.548349 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:48.549153 28717 leader_election.cc:304] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 19:59:48.536623 29933 raft_consensus.cc:2802] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250905 19:59:48.494458 29809 consensus_meta.cc:287] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.594s user 0.008s sys 0.006s
W20250905 19:59:48.553906 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:48.465010 30081 raft_consensus.cc:3058] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:48.556566 30000 raft_consensus.cc:513] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:48.556851 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:48.557888 30063 raft_consensus.cc:2747] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705) with higher term. Message: Invalid argument: T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 2. Current term is 3.
I20250905 19:59:48.560418 28125 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 19:59:48.563256 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 83620.7us
W20250905 19:59:48.564570 30126 log.cc:927] Time spent T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.079s user 0.001s sys 0.000s
I20250905 19:59:48.564844 29854 consensus_queue.cc:1035] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
W20250905 19:59:48.565249 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 111481us
I20250905 19:59:48.536806 30005 raft_consensus.cc:2802] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 19:59:48.566264 29644 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:48.569578 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:48.575181 30072 raft_consensus.cc:491] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250905 19:59:48.555749 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 62281.4us
I20250905 19:59:48.510677 30004 raft_consensus.cc:3053] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:59:48.585606 28076 raft_consensus.cc:2466] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:59:48.586345 28748 raft_consensus.cc:1238] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
W20250905 19:59:48.646941 28007 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 209) took 1567 ms (client timeout 1797 ms). Trace:
W20250905 19:59:48.647719 28007 rpcz_store.cc:269] 0905 19:59:47.079675 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:47.089518 (+ 9843us) service_pool.cc:225] Handling call
0905 19:59:48.646615 (+1557097us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b5400055ad4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:48.646828 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2929786112}
I20250905 19:59:48.587271 29933 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:48.587446 30058 leader_election.cc:290] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:48.589499 30093 raft_consensus.cc:513] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:48.591295 30036 raft_consensus.cc:2802] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:48.553182 28743 raft_consensus.cc:2887] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250905 19:59:48.635461 30080 raft_consensus.cc:2802] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250905 19:59:48.630909 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 70587.5us
I20250905 19:59:48.629667 28084 raft_consensus.cc:2466] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:59:48.601810 30067 raft_consensus.cc:2747] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:48.632814 30005 raft_consensus.cc:491] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:48.542912 30017 raft_consensus.cc:513] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:48.639113 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:48.598801 28004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 206) took 2242 ms. Trace:
I20250905 19:59:48.635136 29924 raft_consensus.cc:513] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:48.668603 28009 raft_consensus.cc:1273] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20250905 19:59:48.878345 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:48.819006 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:49.055729 28125 leader_election.cc:304] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 19:59:49.056295 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:49.056404 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:49.057049 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:49.056875 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:48.739130 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:48.791374 28151 raft_consensus.cc:1238] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20250905 19:59:49.058298 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:48.798072 28154 raft_consensus.cc:1238] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: []
W20250905 19:59:48.894414 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:48.813853 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:48.732877 28122 leader_election.cc:304] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:48.796613 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:48.666831 28051 leader_election.cc:304] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 19:59:49.061197 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:48.636946 30072 raft_consensus.cc:3058] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Advancing to term 5
W20250905 19:59:49.061744 27973 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:48.751241 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 67270.8us
I20250905 19:59:48.638669 30004 raft_consensus.cc:738] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:49.072006 30086 raft_consensus.cc:2802] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 19:59:48.909978 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:49.073627 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:48.661152 29933 raft_consensus.cc:3058] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:49.066414 28004 rpcz_store.cc:276] 0905 19:59:46.311327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:46.938892 (+627565us) service_pool.cc:225] Handling call
0905 19:59:46.939260 (+ 368us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:48.434408 (+1495148us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:48.506637 (+ 72229us) spinlock_profiling.cc:243] Waited 1.43 s on lock 0x7b54001315d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:48.553982 (+ 47345us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3298518912}
I20250905 19:59:48.848964 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:59:48.669881 30079 consensus_meta.cc:287] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.514s user 0.009s sys 0.002s
I20250905 19:59:48.676743 30036 raft_consensus.cc:491] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:49.055030 29995 leader_election.cc:290] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:49.082330 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:49.094318 28154 raft_consensus.cc:2466] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
I20250905 19:59:49.095665 29924 leader_election.cc:290] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:48.665505 30054 leader_election.cc:290] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:49.085407 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:48.773355 29955 raft_consensus.cc:491] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:49.091146 29905 consensus_meta.cc:287] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.615s user 0.011s sys 0.000s
I20250905 19:59:48.921205 29917 raft_consensus.cc:3053] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:49.101917 29917 raft_consensus.cc:738] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:49.084970 30059 raft_consensus.cc:2747] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:49.079584 30086 raft_consensus.cc:491] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:49.103226 30086 raft_consensus.cc:3058] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:49.104319 29995 raft_consensus.cc:491] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:49.104995 29995 raft_consensus.cc:513] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:49.108536 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 19:59:49.109541 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 635181us
I20250905 19:59:49.110551 29995 leader_election.cc:290] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 19:59:49.100018 29871 consensus_meta.cc:287] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.765s user 0.011s sys 0.000s
I20250905 19:59:49.112210 29871 raft_consensus.cc:2747] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:48.891042 29854 raft_consensus.cc:491] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:49.112103 27975 leader_election.cc:304] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:49.113224 29854 raft_consensus.cc:513] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 19:59:49.114471 28153 raft_consensus.cc:2466] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:49.116369 29854 leader_election.cc:290] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:49.117569 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:49.119608 28152 raft_consensus.cc:1273] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250905 19:59:49.080168 30000 leader_election.cc:290] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:48.803663 30093 leader_election.cc:290] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:49.070447 30005 raft_consensus.cc:3058] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:49.139700 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:48.919274 29864 leader_election.cc:290] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:49.121644 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:48.789219 30100 raft_consensus.cc:491] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:49.120383 27894 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet e707b0f561934674947fb463ce626a07 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20250905 19:59:49.103199 29917 consensus_queue.cc:260] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:48.706208 30037 consensus_queue.cc:1035] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.391s
I20250905 19:59:49.125023 29905 raft_consensus.cc:2747] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:49.097208 29941 raft_consensus.cc:2747] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250905 19:59:49.128377 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:48.675963 29906 raft_consensus.cc:513] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:49.134382 30102 consensus_meta.cc:287] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.653s user 0.012s sys 0.000s
I20250905 19:59:48.670190 29826 consensus_queue.cc:1046] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 5, Last known committed idx: 4, Time since last communication: 0.431s
I20250905 19:59:48.674660 29644 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:49.135251 29955 raft_consensus.cc:513] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:48.671054 29943 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:49.136034 30086 raft_consensus.cc:513] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:49.159834 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 495149us
I20250905 19:59:49.160980 30086 leader_election.cc:290] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 19:59:49.097805 30089 consensus_meta.cc:287] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.509s user 0.010s sys 0.001s
I20250905 19:59:49.161298 28122 leader_election.cc:304] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:49.161760 30089 raft_consensus.cc:513] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:48.706530 29987 consensus_queue.cc:1035] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.393s
I20250905 19:59:49.098137 30036 raft_consensus.cc:3058] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:49.130214 27907 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34448 (request call id 253) took 9358 ms. Trace:
W20250905 19:59:49.150552 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 49574.8us
I20250905 19:59:49.174559 28749 raft_consensus.cc:2391] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
W20250905 19:59:49.144155 29934 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.006s sys 0.021s
W20250905 19:59:49.156033 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 493068us
I20250905 19:59:48.742369 29964 consensus_queue.cc:260] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:49.169466 28005 raft_consensus.cc:1238] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
W20250905 19:59:49.172667 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 60790.1us
W20250905 19:59:49.181721 29992 consensus_meta.cc:287] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 1.243s user 0.016s sys 0.002s
I20250905 19:59:49.190344 28122 leader_election.cc:304] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:49.176436 30086 raft_consensus.cc:2747] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250905 19:59:49.174610 29826 consensus_queue.cc:1230] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer 7e9c9f591f05464f9b097bf003d14f4a log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250905 19:59:49.174216 27907 rpcz_store.cc:276] 0905 19:59:39.771350 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:39.771506 (+ 156us) service_pool.cc:225] Handling call
0905 19:59:49.129982 (+9358476us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:59:47.478603 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:59:47.599956 (+121353us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:59:47.603453 (+ 3497us) write_op.cc:435] Acquired schema lock
0905 19:59:47.609877 (+ 6424us) tablet.cc:660] Decoding operations
0905 19:59:47.724037 (+114160us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:59:47.724100 (+ 63us) write_op.cc:641] Partition lock acquired for write op
0905 19:59:47.724127 (+ 27us) tablet.cc:683] Acquiring locks for 5 operations
0905 19:59:47.724828 (+ 701us) tablet.cc:699] Row locks acquired
0905 19:59:47.724848 (+ 20us) write_op.cc:260] PREPARE: finished
0905 19:59:47.724995 (+ 147us) write_op.cc:270] Start()
0905 19:59:47.725108 (+ 113us) write_op.cc:276] Timestamp: P: 1757102387724945 usec, L: 0
0905 19:59:47.725127 (+ 19us) op_driver.cc:348] REPLICATION: starting
0905 19:59:47.746806 (+ 21679us) log.cc:844] Serialized 2085 byte log entry
0905 19:59:48.108679 (+361873us) spinlock_profiling.cc:243] Waited 23.3 ms on lock 0x7b18003351a0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49aca43e2 00007fe49aca67b1 00007fe49aca6769 00007fe49aca66f9 00007fe49aca66c1 00007fe49aca59bd 0000000000379254 0000000000379099 00007fe49ac8776a
0905 19:59:48.367740 (+259061us) op_driver.cc:464] REPLICATION: finished
0905 19:59:48.537290 (+169550us) write_op.cc:301] APPLY: starting
0905 19:59:48.537406 (+ 116us) tablet.cc:1365] starting BulkCheckPresence
0905 19:59:48.537571 (+ 165us) tablet.cc:1368] finished BulkCheckPresence
0905 19:59:48.537605 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:59:48.538250 (+ 645us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:59:48.538543 (+ 293us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0905 19:59:48.538607 (+ 64us) write_op.cc:312] APPLY: finished
0905 19:59:48.538909 (+ 302us) log.cc:844] Serialized 47 byte log entry
0905 19:59:48.539093 (+ 184us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:59:48.539213 (+ 120us) write_op.cc:454] Released schema lock
0905 19:59:48.539260 (+ 47us) write_op.cc:341] FINISH: Updating metrics
0905 19:59:48.539789 (+ 529us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:59:48.539828 (+ 39us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":27376,"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":169212,"apply.run_cpu_time_us":2425,"apply.run_wall_time_us":2425,"num_ops":5,"prepare.queue_time_us":312688,"prepare.run_cpu_time_us":45506,"prepare.run_wall_time_us":618926,"raft.queue_time_us":85609,"raft.run_cpu_time_us":54979,"raft.run_wall_time_us":78497,"replication_time_us":642217,"spinlock_wait_cycles":53681408,"thread_start_us":551956,"threads_started":4,"wal-append.queue_time_us":268508}]]}
I20250905 19:59:49.153970 30100 raft_consensus.cc:513] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:49.205240 29934 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.639s user 0.007s sys 0.021s
I20250905 19:59:49.218956 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:49.236148 28051 leader_election.cc:304] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:49.184897 29905 consensus_queue.cc:1046] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
I20250905 19:59:49.174046 29924 consensus_queue.cc:1046] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
W20250905 19:59:49.246026 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 81456.1us
I20250905 19:59:49.243607 29905 consensus_queue.cc:1230] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Peer e4413ae5e0b64b52a6f360ac580239b9 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250905 19:59:49.250078 29840 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:49.239670 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:49.244570 29964 raft_consensus.cc:3058] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:49.236089 28747 raft_consensus.cc:2466] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
I20250905 19:59:49.263762 28051 leader_election.cc:304] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:49.250559 29848 raft_consensus.cc:491] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:49.266599 30091 consensus_meta.cc:287] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.717s user 0.004s sys 0.008s
I20250905 19:59:49.245455 30079 raft_consensus.cc:2747] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250905 19:59:49.273871 30081 consensus_meta.cc:287] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.644s user 0.010s sys 0.001s
W20250905 19:59:49.283140 30128 log.cc:927] Time spent T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20250905 19:59:49.286010 28122 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:49.251125 29826 raft_consensus.cc:2747] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:49.262223 30017 leader_election.cc:290] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 19:59:49.320246 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:49.337214 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 107146us
I20250905 19:59:49.339560 27975 leader_election.cc:304] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:49.306885 29941 raft_consensus.cc:2747] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:49.314215 30017 raft_consensus.cc:491] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:49.347846 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 84206.7us
I20250905 19:59:49.312870 30080 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:49.329748 29906 leader_election.cc:290] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:49.304534 30091 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:49.301991 29848 raft_consensus.cc:513] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:49.292122 29840 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:49.341411 29644 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:49.357896 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 19:59:49.358815 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:49.346884 30089 leader_election.cc:290] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:49.349534 30079 raft_consensus.cc:2747] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 19:59:49.349140 29943 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:49.371169 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:49.354517 30080 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Advancing to term 7
I20250905 19:59:49.375025 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:49.358462 29955 leader_election.cc:290] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:49.346207 29933 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:49.377239 28749 raft_consensus.cc:2466] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
I20250905 19:59:49.381137 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:49.376964 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:49.382128 28156 raft_consensus.cc:2391] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 5: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:49.354094 30017 raft_consensus.cc:513] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:49.386397 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 100479us
I20250905 19:59:49.371476 29871 raft_consensus.cc:491] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:49.388561 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:49.389400 28154 raft_consensus.cc:3058] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:49.376916 29987 raft_consensus.cc:491] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:49.393128 29871 raft_consensus.cc:513] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:49.393491 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:49.394025 29987 raft_consensus.cc:513] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:49.390861 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:49.398142 28743 raft_consensus.cc:2391] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 5: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 19:59:49.398023 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:49.401822 30017 leader_election.cc:290] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:49.404330 28005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1308) took 1262 ms. Trace:
I20250905 19:59:49.391258 28152 raft_consensus.cc:1273] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250905 19:59:49.378827 29644 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:49.404642 28005 rpcz_store.cc:276] 0905 19:59:48.142280 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:48.325215 (+182935us) service_pool.cc:225] Handling call
0905 19:59:48.325657 (+ 442us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:49.213244 (+887587us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:49.404055 (+190811us) spinlock_profiling.cc:243] Waited 843 ms on lock 0x7b540034ab54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:49.404266 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1939798272}
I20250905 19:59:49.372236 30072 raft_consensus.cc:513] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:49.420126 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 80659us
I20250905 19:59:49.386504 29992 raft_consensus.cc:491] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:49.432032 28747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40918 (request call id 476) took 1302 ms. Trace:
I20250905 19:59:49.421447 30017 consensus_queue.cc:1035] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20250905 19:59:49.391307 30089 consensus_queue.cc:1035] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20250905 19:59:49.406235 29826 raft_consensus.cc:491] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:49.407694 29987 leader_election.cc:290] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:49.433789 29917 raft_consensus.cc:3058] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:49.434123 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:49.434808 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:49.436892 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:49.437731 29871 leader_election.cc:290] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:49.438572 28154 raft_consensus.cc:2466] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 3.
W20250905 19:59:49.440829 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:49.441622 28122 leader_election.cc:304] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:49.438283 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:49.472015 28750 raft_consensus.cc:3058] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Advancing to term 7
I20250905 19:59:49.438611 30005 raft_consensus.cc:513] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:49.443636 30134 log.cc:927] Time spent T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20250905 19:59:49.444690 28743 raft_consensus.cc:1273] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250905 19:59:49.474795 28717 leader_election.cc:304] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
W20250905 19:59:49.445221 30135 log.cc:927] Time spent T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.070s user 0.000s sys 0.001s
I20250905 19:59:49.392204 30100 leader_election.cc:290] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:49.451004 28751 raft_consensus.cc:2391] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 5: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
W20250905 19:59:49.461969 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:49.430262 29943 raft_consensus.cc:2747] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250905 19:59:49.464311 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:49.468119 28752 raft_consensus.cc:2391] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 3: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
W20250905 19:59:49.470513 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 198481us
W20250905 19:59:49.483002 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 105633us
I20250905 19:59:49.486374 28051 leader_election.cc:304] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:49.488898 28051 leader_election.cc:304] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:49.489656 29884 raft_consensus.cc:491] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:49.451370 29848 leader_election.cc:290] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:49.494035 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:49.473307 30140 raft_consensus.cc:491] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:49.475628 30091 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:49.448437 29644 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:49.517251 29644 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:49.505733 29955 raft_consensus.cc:2747] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 19:59:49.473307 28748 raft_consensus.cc:2466] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 19:59:49.520491 30036 raft_consensus.cc:513] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:49.521013 30136 raft_consensus.cc:491] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:49.521749 28049 leader_election.cc:304] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:49.441769 28747 rpcz_store.cc:276] 0905 19:59:48.129169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:48.131354 (+ 2185us) service_pool.cc:225] Handling call
0905 19:59:49.431799 (+1300445us) spinlock_profiling.cc:243] Waited 997 ms on lock 0x7b540053cd54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:49.431938 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2292702080}
W20250905 19:59:49.510587 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:49.510772 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:49.506393 30005 leader_election.cc:290] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:49.528117 28158 raft_consensus.cc:3058] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:49.528880 28049 leader_election.cc:304] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:49.530642 29941 raft_consensus.cc:2802] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 19:59:49.531095 29941 raft_consensus.cc:491] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:49.531424 29941 raft_consensus.cc:3058] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:49.494415 30141 raft_consensus.cc:2802] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:49.532634 30141 raft_consensus.cc:695] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:49.498251 29871 raft_consensus.cc:2747] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250905 19:59:49.511399 29840 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:49.512341 30138 log.cc:927] Time spent T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.091s user 0.001s sys 0.000s
W20250905 19:59:49.510859 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 90722.8us
I20250905 19:59:49.500666 30100 raft_consensus.cc:3058] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:49.513736 27973 leader_election.cc:304] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:49.539510 27973 leader_election.cc:304] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:49.541723 29840 consensus_queue.cc:1035] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250905 19:59:49.542629 29906 raft_consensus.cc:2802] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:49.543210 29840 consensus_queue.cc:1035] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250905 19:59:49.514403 29943 consensus_queue.cc:1035] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:49.521847 30136 raft_consensus.cc:513] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:49.503161 28744 raft_consensus.cc:1238] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 4. Current term is 5. Ops: []
I20250905 19:59:49.502377 28157 raft_consensus.cc:2391] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 5: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:49.470175 29992 raft_consensus.cc:513] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:49.473876 29987 consensus_queue.cc:1035] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20250905 19:59:49.546649 30136 leader_election.cc:290] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:49.524461 29884 raft_consensus.cc:513] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:49.525556 27974 leader_election.cc:304] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:49.474092 29826 raft_consensus.cc:513] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:49.549263 29884 leader_election.cc:290] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:49.549804 29884 raft_consensus.cc:2802] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 19:59:49.550009 29826 leader_election.cc:290] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:49.526276 29955 raft_consensus.cc:2802] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 19:59:49.550653 29955 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:49.551092 29955 raft_consensus.cc:3058] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:49.552152 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:49.553013 28743 raft_consensus.cc:2391] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 5: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 19:59:49.525947 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:49.526733 28746 raft_consensus.cc:1215] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Deduplicated request from leader. Original: 1.4->[3.5-3.5] Dedup: 3.5->[]
I20250905 19:59:49.513386 29933 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:49.543443 30017 raft_consensus.cc:2802] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:49.554864 28746 raft_consensus.cc:1238] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20250905 19:59:49.555074 30017 raft_consensus.cc:491] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:49.533993 30141 consensus_queue.cc:237] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:49.555493 30017 raft_consensus.cc:3058] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:49.511739 30004 consensus_queue.cc:260] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:49.524101 30140 raft_consensus.cc:513] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:49.558904 30036 leader_election.cc:290] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:49.558944 30004 raft_consensus.cc:3058] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:49.560303 30140 leader_election.cc:290] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:49.547509 29992 leader_election.cc:290] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:49.531343 30072 leader_election.cc:290] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:49.561728 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:49.543131 29906 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:49.566458 29906 raft_consensus.cc:3058] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
W20250905 19:59:49.568946 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 49074.3us
I20250905 19:59:49.569078 29848 consensus_queue.cc:1035] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250905 19:59:49.567764 27975 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:49.575733 29943 raft_consensus.cc:2747] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250905 19:59:49.575866 28050 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:49.576629 29826 consensus_queue.cc:1046] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 19:59:49.577443 28050 leader_election.cc:304] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:49.577807 29853 raft_consensus.cc:3053] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:59:49.577437 29826 raft_consensus.cc:2747] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250905 19:59:49.578269 29853 raft_consensus.cc:738] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:49.579195 29853 consensus_queue.cc:260] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:49.580392 29853 raft_consensus.cc:3058] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:49.580274 29943 consensus_queue.cc:1046] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20250905 19:59:49.582237 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:49.583356 28748 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
W20250905 19:59:49.583885 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:49.584676 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:49.585805 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:49.586619 28748 raft_consensus.cc:2391] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
W20250905 19:59:49.586791 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:49.600528 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:49.600879 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:49.601402 28157 raft_consensus.cc:2466] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 3.
I20250905 19:59:49.601511 28150 raft_consensus.cc:3058] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 3
I20250905 19:59:49.604774 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:49.605489 28152 raft_consensus.cc:2466] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 19:59:49.550184 29884 raft_consensus.cc:491] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:49.606762 29884 raft_consensus.cc:3058] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:49.495205 29924 consensus_queue.cc:1230] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer e4413ae5e0b64b52a6f360ac580239b9 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20250905 19:59:49.608266 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:49.609920 28750 raft_consensus.cc:2466] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 7.
I20250905 19:59:49.612005 28150 raft_consensus.cc:2466] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 3.
I20250905 19:59:49.610483 28050 leader_election.cc:304] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 19:59:49.620227 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 52643.5us
I20250905 19:59:49.621151 28717 leader_election.cc:304] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 19:59:49.624158 28158 raft_consensus.cc:2466] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 19:59:49.622664 28717 leader_election.cc:304] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 19:59:49.630156 28717 leader_election.cc:304] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 19:59:49.632447 28006 raft_consensus.cc:1273] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250905 19:59:49.636701 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:49.637753 28010 raft_consensus.cc:2391] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:49.607821 30067 consensus_queue.cc:1046] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20250905 19:59:49.622648 29864 raft_consensus.cc:2802] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:49.645762 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:49.638316 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:49.627094 30072 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:49.635851 28717 leader_election.cc:304] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 19:59:49.633515 30140 raft_consensus.cc:2802] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250905 19:59:49.665189 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:49.659332 28004 raft_consensus.cc:2374] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5. Re-sending same reply.
W20250905 19:59:49.672888 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:49.668195 30080 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:49.634601 30036 raft_consensus.cc:2802] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 19:59:49.670974 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:49.667433 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:49.669781 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:49.688364 28002 raft_consensus.cc:3058] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
W20250905 19:59:49.680480 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:49.691643 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:49.692539 28156 raft_consensus.cc:2391] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 19:59:49.650240 29840 raft_consensus.cc:2802] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:49.647050 29995 raft_consensus.cc:2802] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:49.648123 30100 raft_consensus.cc:2747] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:49.694254 28009 raft_consensus.cc:1273] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250905 19:59:49.702186 27975 leader_election.cc:304] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:49.696195 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:49.696792 28050 leader_election.cc:304] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:49.691022 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:49.672457 30140 raft_consensus.cc:491] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:49.713007 28158 raft_consensus.cc:1273] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:59:49.660694 29864 raft_consensus.cc:491] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:49.705632 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:49.697328 29992 raft_consensus.cc:2802] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader election won for term 5
I20250905 19:59:49.710405 28083 raft_consensus.cc:2466] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 19:59:49.705518 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:49.691298 29848 raft_consensus.cc:3053] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:49.686385 30036 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250905 19:59:49.683247 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 47472.5us
I20250905 19:59:49.718243 28084 raft_consensus.cc:2408] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250905 19:59:49.705837 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:49.720628 28152 raft_consensus.cc:2391] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 5: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:49.701911 29995 raft_consensus.cc:491] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:49.707001 30063 raft_consensus.cc:3053] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Stepping down as leader of term 4
I20250905 19:59:49.727156 27974 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:49.708341 29840 raft_consensus.cc:695] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 3 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:49.705132 29955 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:49.718578 30140 raft_consensus.cc:3058] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:49.720016 29864 raft_consensus.cc:3058] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:49.737624 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:49.755865 28009 raft_consensus.cc:3058] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:49.719758 30067 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:49.735499 30036 raft_consensus.cc:3058] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Advancing to term 6
I20250905 19:59:49.723215 29848 raft_consensus.cc:738] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
W20250905 19:59:49.744199 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:49.736462 29917 raft_consensus.cc:2747] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:49.738495 29995 raft_consensus.cc:3058] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:49.766676 28750 raft_consensus.cc:1273] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250905 19:59:49.747711 29924 raft_consensus.cc:2802] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader election won for term 7
I20250905 19:59:49.737584 30063 raft_consensus.cc:738] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:49.743930 29644 raft_consensus.cc:3058] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:49.718153 29941 raft_consensus.cc:513] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:49.744964 29992 raft_consensus.cc:695] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 5 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:49.779237 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:49.780100 28006 raft_consensus.cc:2466] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 2.
W20250905 19:59:49.770035 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 109584us
W20250905 19:59:49.777189 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 80475.3us
I20250905 19:59:49.715323 30072 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:49.781639 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:59:49.789847 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:49.807363 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:49.780728 29924 raft_consensus.cc:695] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:49.773185 27907 catalog_manager.cc:5582] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 1 to 4, leader changed from e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) to 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132). New cstate: current_term: 4 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 19:59:49.816241 28002 raft_consensus.cc:2466] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 19:59:49.817459 27975 leader_election.cc:304] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:49.786926 30093 raft_consensus.cc:491] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:49.805531 29964 consensus_meta.cc:287] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.510s user 0.012s sys 0.000s
I20250905 19:59:49.833249 28155 raft_consensus.cc:1273] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:59:49.835843 29943 raft_consensus.cc:2747] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 19:59:49.840082 29924 consensus_queue.cc:237] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:49.840682 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 43679.2us
W20250905 19:59:49.819782 28122 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 2.785s (SENT) [suppressed 63 similar messages]
I20250905 19:59:49.844329 27975 leader_election.cc:304] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
W20250905 19:59:49.845486 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 356780us
I20250905 19:59:49.845978 29917 raft_consensus.cc:2747] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250905 19:59:49.846513 29943 consensus_queue.cc:1046] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:49.785883 30079 consensus_queue.cc:1035] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20250905 19:59:49.877688 28007 raft_consensus.cc:2466] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 19:59:49.782006 30067 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:49.878611 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 58825.4us
I20250905 19:59:49.832347 29840 consensus_queue.cc:237] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:49.881171 27975 leader_election.cc:304] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:49.809214 30017 raft_consensus.cc:513] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:49.836884 29826 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:49.837054 27907 catalog_manager.cc:5582] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 1 to 3, leader changed from cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) to 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132). New cstate: current_term: 3 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 19:59:49.850461 28159 raft_consensus.cc:2466] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:49.817425 30165 consensus_queue.cc:1035] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.066s
I20250905 19:59:49.897686 27907 catalog_manager.cc:5582] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 19:59:49.902727 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:49.903537 28150 raft_consensus.cc:2391] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 5: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:49.885628 30166 raft_consensus.cc:491] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:49.835749 30093 raft_consensus.cc:513] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:49.906957 28156 raft_consensus.cc:1238] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 6. Current term is 7. Ops: []
I20250905 19:59:49.911295 28008 raft_consensus.cc:1238] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:49.904259 27907 catalog_manager.cc:5582] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: config changed from index 3 to 4, term changed from 1 to 2, leader changed from e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) to 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132), VOTER 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) evicted. New cstate: current_term: 2 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 19:59:49.895335 30102 consensus_queue.cc:1035] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20250905 19:59:49.926685 27907 catalog_manager.cc:5582] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 2 to 4, leader changed from 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 4 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
W20250905 19:59:49.928594 30160 log.cc:927] Time spent T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.123s user 0.000s sys 0.001s
I20250905 19:59:49.928829 29905 raft_consensus.cc:3053] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:49.928927 30102 consensus_queue.cc:1046] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 19:59:49.929419 29905 raft_consensus.cc:738] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:49.932222 29941 leader_election.cc:290] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:49.927110 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:49.939867 28155 raft_consensus.cc:3058] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:49.876991 30063 consensus_queue.cc:260] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:49.913522 29955 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:49.897658 29826 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:49.883798 29992 consensus_queue.cc:237] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:49.944841 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 63762.1us
I20250905 19:59:49.819588 29848 consensus_queue.cc:260] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:49.938010 30149 log.cc:927] Time spent T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.164s user 0.001s sys 0.000s
I20250905 19:59:49.946924 27907 catalog_manager.cc:5582] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 1 to 3, leader changed from e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) to 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132). New cstate: current_term: 3 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250905 19:59:49.948650 30162 log.cc:927] Time spent T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.099s user 0.001s sys 0.000s
I20250905 19:59:49.930346 29905 consensus_queue.cc:260] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:49.949844 30153 log.cc:927] Time spent T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.120s user 0.001s sys 0.000s
I20250905 19:59:49.953040 29644 raft_consensus.cc:2747] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:49.955090 30067 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:49.955178 29906 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:49.956197 30063 raft_consensus.cc:3058] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:49.956337 28051 leader_election.cc:304] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:49.963230 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:49.986886 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 19:59:49.987211 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:49.987617 28003 raft_consensus.cc:3058] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:49.925186 30166 raft_consensus.cc:513] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:49.968061 28009 raft_consensus.cc:2466] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 19:59:49.954001 30036 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:49.977015 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:49.959141 30080 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 19:59:49.979959 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:49.961561 30172 raft_consensus.cc:2747] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250905 19:59:50.004163 29848 raft_consensus.cc:3058] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:49.985237 27975 leader_election.cc:304] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:49.998126 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:59:50.012681 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:50.012912 28150 raft_consensus.cc:2466] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
I20250905 19:59:49.998624 28750 raft_consensus.cc:1238] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:50.001070 30036 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:50.013974 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:50.017234 28050 leader_election.cc:304] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:50.020810 28003 raft_consensus.cc:2466] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
W20250905 19:59:50.023084 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 132279us
W20250905 19:59:50.026728 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:49.996440 29955 raft_consensus.cc:2747] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:50.051684 28050 leader_election.cc:304] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:50.023990 30165 raft_consensus.cc:2802] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 19:59:50.063041 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:50.064072 28750 raft_consensus.cc:2366] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 5. Current term is 6.
I20250905 19:59:50.064782 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:50.065271 28049 leader_election.cc:400] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Vote denied by peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299) with higher term. Message: Invalid argument: T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 5. Current term is 6.
I20250905 19:59:50.065402 28150 raft_consensus.cc:3058] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
I20250905 19:59:50.065640 28049 leader_election.cc:403] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
W20250905 19:59:50.066524 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:50.025794 29826 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:50.047315 30172 raft_consensus.cc:2747] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250905 19:59:50.071853 30102 raft_consensus.cc:2802] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250905 19:59:50.077250 30167 log.cc:927] Time spent T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.106s user 0.001s sys 0.000s
I20250905 19:59:50.072043 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:50.075038 29941 raft_consensus.cc:3058] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
W20250905 19:59:50.099041 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:50.099668 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:50.082237 28009 raft_consensus.cc:1273] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250905 19:59:50.055356 30017 leader_election.cc:290] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:50.095950 28748 raft_consensus.cc:3058] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:50.088198 30102 raft_consensus.cc:491] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:50.098297 28152 raft_consensus.cc:1273] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250905 19:59:50.134629 30102 raft_consensus.cc:3058] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:50.136283 29826 raft_consensus.cc:491] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:50.136873 29826 raft_consensus.cc:513] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:50.070688 30165 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:50.151661 30155 consensus_queue.cc:1035] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20250905 19:59:50.155184 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:50.132187 30172 raft_consensus.cc:491] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:50.167753 30172 raft_consensus.cc:513] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:50.150008 30165 raft_consensus.cc:3058] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
W20250905 19:59:50.190959 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:50.217720 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 246014us
I20250905 19:59:50.220922 28155 raft_consensus.cc:2466] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
I20250905 19:59:50.227778 28050 leader_election.cc:304] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:50.223263 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 19:59:50.222245 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 265991us
I20250905 19:59:50.232051 28751 raft_consensus.cc:2466] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
W20250905 19:59:50.255498 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 190364us
W20250905 19:59:50.263756 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:50.269012 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:50.222572 29905 raft_consensus.cc:3058] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:50.274008 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:50.276654 28051 leader_election.cc:304] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 19:59:50.263132 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:50.243058 29853 raft_consensus.cc:2802] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election won for term 5
I20250905 19:59:50.280933 28151 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 3.
I20250905 19:59:50.288306 28150 raft_consensus.cc:2466] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6.
I20250905 19:59:50.267841 30093 leader_election.cc:290] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:50.264315 30072 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:50.263305 30081 raft_consensus.cc:491] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:50.318652 27975 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:50.321983 28007 raft_consensus.cc:2466] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:50.267107 30170 raft_consensus.cc:491] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:50.323146 28717 leader_election.cc:304] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 19:59:50.326754 30166 leader_election.cc:290] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:50.328665 30063 raft_consensus.cc:2747] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250905 19:59:50.328851 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 46423.2us
I20250905 19:59:50.330495 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 19:59:50.335263 29995 consensus_meta.cc:287] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.558s user 0.020s sys 0.010s
W20250905 19:59:50.335938 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 59423.9us
I20250905 19:59:50.335829 29995 raft_consensus.cc:513] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 19:59:50.338407 30170 raft_consensus.cc:513] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:50.322540 29906 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:50.320890 29853 raft_consensus.cc:695] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 5 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:50.313071 30172 leader_election.cc:290] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:50.351900 28006 raft_consensus.cc:3058] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:50.354722 29995 leader_election.cc:290] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421)
I20250905 19:59:50.355549 27975 leader_election.cc:304] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
W20250905 19:59:50.355916 29884 consensus_meta.cc:287] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.748s user 0.018s sys 0.003s
I20250905 19:59:50.356554 30141 raft_consensus.cc:2802] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader election won for term 6
I20250905 19:59:50.360880 30141 raft_consensus.cc:695] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 6 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:50.360711 29884 raft_consensus.cc:513] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:50.362534 28005 raft_consensus.cc:2391] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:50.362936 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 19:59:50.364638 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 45161.4us
I20250905 19:59:50.367393 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:50.368517 28751 raft_consensus.cc:2391] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 2: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 19:59:50.370605 29644 raft_consensus.cc:2747] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250905 19:59:50.372671 28051 leader_election.cc:304] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:50.378661 29955 raft_consensus.cc:2747] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:50.362037 30141 consensus_queue.cc:237] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:50.374073 28051 leader_election.cc:304] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:50.389562 28155 raft_consensus.cc:2466] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:50.391438 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:50.392875 28007 raft_consensus.cc:1238] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:50.397730 28050 leader_election.cc:304] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:50.317195 29826 leader_election.cc:290] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:50.411531 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:50.409330 27975 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:50.379560 30172 raft_consensus.cc:2747] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 19:59:50.424675 28151 raft_consensus.cc:2466] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
W20250905 19:59:50.391424 29864 consensus_meta.cc:287] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.622s user 0.010s sys 0.011s
I20250905 19:59:50.440884 29995 raft_consensus.cc:2802] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:50.441547 29995 raft_consensus.cc:695] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:50.442561 29995 consensus_queue.cc:237] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:50.391811 29853 consensus_queue.cc:237] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:50.445053 30170 leader_election.cc:290] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:50.426524 29955 raft_consensus.cc:2802] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:50.446336 29955 raft_consensus.cc:695] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:50.441397 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:50.448040 28007 raft_consensus.cc:2466] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
W20250905 19:59:50.449652 30140 consensus_meta.cc:287] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.682s user 0.009s sys 0.010s
I20250905 19:59:50.456497 28050 leader_election.cc:304] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:50.465245 30063 raft_consensus.cc:2747] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250905 19:59:50.468668 28155 raft_consensus.cc:1273] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250905 19:59:50.474606 28006 raft_consensus.cc:2466] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 19:59:50.474872 30102 raft_consensus.cc:513] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:50.475548 29884 leader_election.cc:290] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:50.424247 29964 raft_consensus.cc:491] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:50.477283 29964 raft_consensus.cc:513] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:50.493384 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:50.545395 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:50.550925 28718 leader_election.cc:304] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 19:59:50.552428 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:50.334586 30081 raft_consensus.cc:513] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:50.525458 29964 leader_election.cc:290] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:50.586273 30081 leader_election.cc:290] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:50.450109 30140 raft_consensus.cc:513] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:50.622205 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:50.626610 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 73016.6us
W20250905 19:59:50.630019 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:50.614991 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:50.615646 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:50.635918 28083 raft_consensus.cc:2408] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250905 19:59:50.636889 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:50.640522 28084 raft_consensus.cc:3058] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:50.639254 28125 leader_election.cc:304] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 19:59:50.641562 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:50.642673 28003 raft_consensus.cc:1238] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:50.554066 30170 raft_consensus.cc:2802] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 19:59:50.639304 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:50.637311 28008 raft_consensus.cc:1273] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20250905 19:59:50.644711 27973 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:50.644982 28125 leader_election.cc:304] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:50.651495 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:59:50.647265 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:50.458684 29864 raft_consensus.cc:513] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:50.634055 28748 consensus_meta.cc:287] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.501s user 0.020s sys 0.032s
I20250905 19:59:50.660956 28748 raft_consensus.cc:2466] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
I20250905 19:59:50.662213 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:50.648459 30176 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:50.662946 28156 raft_consensus.cc:2466] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
I20250905 19:59:50.663246 30176 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:50.648337 30036 consensus_queue.cc:1035] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.016s
I20250905 19:59:50.665341 30176 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:50.668001 29864 leader_election.cc:290] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:50.648284 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:50.669127 28004 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Advancing to term 7
I20250905 19:59:50.669191 28080 raft_consensus.cc:1273] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250905 19:59:50.652786 27973 leader_election.cc:304] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:50.648795 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:50.644953 28718 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:50.675311 28084 raft_consensus.cc:2466] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 19:59:50.675541 28009 raft_consensus.cc:1273] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
W20250905 19:59:50.655467 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 281456us
I20250905 19:59:50.587316 29992 raft_consensus.cc:2747] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 19:59:50.624218 30136 raft_consensus.cc:3053] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 LEADER]: Stepping down as leader of term 6
I20250905 19:59:50.659727 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:50.656677 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:50.673473 28007 raft_consensus.cc:2466] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 19:59:50.672773 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:50.679029 28751 raft_consensus.cc:3058] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:50.679193 28081 raft_consensus.cc:3058] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:50.677982 28752 raft_consensus.cc:1273] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20250905 19:59:50.680266 28715 leader_election.cc:336] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.785s (SENT)
I20250905 19:59:50.680770 28715 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:50.681030 28002 raft_consensus.cc:2391] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 5: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 19:59:50.681339 28050 leader_election.cc:304] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 19:59:50.681784 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:50.679409 27973 leader_election.cc:304] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:50.689287 27973 leader_election.cc:304] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:50.690389 27975 leader_election.cc:304] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
W20250905 19:59:50.693238 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 48452us
I20250905 19:59:50.693917 28718 leader_election.cc:304] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:50.646636 30063 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:50.648962 29848 raft_consensus.cc:2747] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:50.701138 28008 raft_consensus.cc:1273] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250905 19:59:50.650164 29905 raft_consensus.cc:491] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:50.669143 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:50.710067 28079 raft_consensus.cc:3058] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:50.596943 30072 consensus_queue.cc:1035] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.090s
I20250905 19:59:50.712541 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:50.698805 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:50.705243 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:50.649701 30170 raft_consensus.cc:491] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:50.722744 30170 raft_consensus.cc:3058] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:50.724509 30140 leader_election.cc:290] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:50.725217 28051 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:50.726506 30063 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:50.727296 29955 consensus_queue.cc:237] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 19:59:50.712291 29943 raft_consensus.cc:2747] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 19:59:50.646359 30005 consensus_queue.cc:1035] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.057s
I20250905 19:59:50.729367 30063 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:50.705173 29840 raft_consensus.cc:2802] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader election won for term 5
I20250905 19:59:50.651750 29964 consensus_queue.cc:1046] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.058s
I20250905 19:59:50.697919 30079 raft_consensus.cc:491] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:50.699242 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:50.731029 29992 raft_consensus.cc:2802] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 19:59:50.707914 29924 raft_consensus.cc:2802] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:50.731559 29992 raft_consensus.cc:491] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:50.731272 30079 raft_consensus.cc:513] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:50.731995 29924 raft_consensus.cc:695] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:50.733093 29924 consensus_queue.cc:237] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:50.733717 30079 leader_election.cc:290] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:50.695706 30148 consensus_queue.cc:1035] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20250905 19:59:50.706238 30106 raft_consensus.cc:2802] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:50.727552 29905 raft_consensus.cc:513] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:50.735260 30106 raft_consensus.cc:695] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:50.697971 30017 raft_consensus.cc:2747] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 19:59:50.736177 30106 consensus_queue.cc:237] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:50.737308 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:50.737818 28718 leader_election.cc:304] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:50.738464 28002 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 3.
W20250905 19:59:50.738905 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 58668.9us
I20250905 19:59:50.706395 30081 raft_consensus.cc:491] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:50.739907 30036 raft_consensus.cc:2802] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader election won for term 5
I20250905 19:59:50.740357 29840 raft_consensus.cc:695] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 5 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:50.740317 30081 raft_consensus.cc:513] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:50.695400 29864 raft_consensus.cc:2802] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:50.737933 28005 raft_consensus.cc:2466] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:50.677937 30176 raft_consensus.cc:2747] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:50.736616 29905 leader_election.cc:290] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:50.720588 30165 consensus_meta.cc:287] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.540s user 0.013s sys 0.000s
W20250905 19:59:50.719538 29941 consensus_meta.cc:287] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.623s user 0.018s sys 0.000s
I20250905 19:59:50.734274 29644 raft_consensus.cc:2747] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 19:59:50.732065 29992 raft_consensus.cc:3058] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:50.754276 30165 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:50.761426 30165 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:50.678634 30136 raft_consensus.cc:738] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:50.743213 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:50.750663 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:50.768680 28084 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 7.
I20250905 19:59:50.750835 28004 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
I20250905 19:59:50.748589 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:50.713462 28750 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 7.
I20250905 19:59:50.755739 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:50.756978 28051 leader_election.cc:304] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:50.740451 30036 raft_consensus.cc:695] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 5 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
W20250905 19:59:50.763967 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 112372us
I20250905 19:59:50.771025 28008 raft_consensus.cc:2466] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 19:59:50.772544 28009 raft_consensus.cc:2466] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 2.
I20250905 19:59:50.772763 27974 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:50.744735 29864 raft_consensus.cc:491] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:50.704839 29906 consensus_queue.cc:1035] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20250905 19:59:50.709491 30080 consensus_queue.cc:1035] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20250905 19:59:50.763127 28050 leader_election.cc:304] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:50.785508 28081 raft_consensus.cc:2466] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
W20250905 19:59:50.833197 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 78233.5us
I20250905 19:59:50.836670 27973 leader_election.cc:304] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:50.838459 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:50.839403 28081 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:59:50.793533 28006 raft_consensus.cc:1273] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250905 19:59:50.779459 28751 raft_consensus.cc:2466] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
W20250905 19:59:50.819713 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:50.754484 29941 raft_consensus.cc:2747] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299) with higher term. Message: Invalid argument: T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 5. Current term is 6.
I20250905 19:59:50.780870 30079 raft_consensus.cc:491] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:50.779596 30155 raft_consensus.cc:2747] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250905 19:59:50.795356 30017 raft_consensus.cc:2802] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 19:59:50.791611 29864 raft_consensus.cc:3058] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:50.848270 27975 leader_election.cc:304] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
W20250905 19:59:50.845044 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:50.791436 28748 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:59:50.856528 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:50.842458 30176 raft_consensus.cc:2802] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election won for term 5
I20250905 19:59:50.853003 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:50.862277 28158 raft_consensus.cc:3058] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
I20250905 19:59:50.846804 30166 raft_consensus.cc:2747] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250905 19:59:50.879184 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 122293us
I20250905 19:59:50.858934 30017 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:50.872121 29905 raft_consensus.cc:2747] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 19:59:50.877287 27974 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:50.855561 30136 consensus_queue.cc:260] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:50.851778 29644 raft_consensus.cc:3058] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
W20250905 19:59:50.918962 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:50.926143 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 48978.7us
W20250905 19:59:50.927536 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 47327.4us
W20250905 19:59:50.928800 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 191034us
W20250905 19:59:50.935277 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 92641.7us
I20250905 19:59:50.941936 28718 leader_election.cc:304] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:50.852447 30079 raft_consensus.cc:513] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:50.938824 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 222394us
I20250905 19:59:50.926544 30017 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Advancing to term 8
I20250905 19:59:50.959553 28125 leader_election.cc:304] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:50.843263 29840 consensus_queue.cc:237] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:50.881300 30176 raft_consensus.cc:695] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:50.934135 29906 raft_consensus.cc:2802] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:50.961855 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:50.965358 28125 leader_election.cc:304] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:50.971088 28009 raft_consensus.cc:3058] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 4
W20250905 19:59:50.976666 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:50.979538 28125 leader_election.cc:304] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:50.981976 28006 raft_consensus.cc:1273] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250905 19:59:50.988106 28158 raft_consensus.cc:2466] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 6.
I20250905 19:59:50.989487 28050 leader_election.cc:304] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:50.976291 29906 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:51.001879 29906 raft_consensus.cc:3058] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:50.969339 30185 raft_consensus.cc:2802] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250905 19:59:51.007038 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 51172.1us
I20250905 19:59:50.990686 30080 raft_consensus.cc:2802] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250905 19:59:51.019624 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:50.995302 30136 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Advancing to term 7
I20250905 19:59:50.992245 30063 raft_consensus.cc:491] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:50.960597 29941 consensus_queue.cc:1035] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20250905 19:59:50.992892 30081 leader_election.cc:290] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:50.974625 30189 consensus_queue.cc:1046] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.041s
I20250905 19:59:51.003302 30196 raft_consensus.cc:2802] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election won for term 6
I20250905 19:59:50.948172 30036 consensus_queue.cc:237] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:51.010716 30185 raft_consensus.cc:491] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250905 19:59:51.070142 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:51.077286 28009 raft_consensus.cc:2466] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
W20250905 19:59:51.080745 30202 log.cc:927] Time spent T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
W20250905 19:59:51.081569 30199 log.cc:927] Time spent T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.220s user 0.000s sys 0.001s
I20250905 19:59:51.082049 30213 consensus_queue.cc:1046] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 19:59:51.082994 30189 raft_consensus.cc:3053] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Stepping down as leader of term 3
W20250905 19:59:51.089419 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:51.104446 30102 leader_election.cc:290] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:51.113427 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 69344.1us
I20250905 19:59:51.021788 30072 raft_consensus.cc:2802] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:51.117619 28051 leader_election.cc:304] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:51.023167 30080 raft_consensus.cc:491] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:51.130383 28079 raft_consensus.cc:2466] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
W20250905 19:59:51.135879 27973 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:51.161496 27973 leader_election.cc:304] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:51.015462 30166 consensus_queue.cc:1035] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.101s
W20250905 19:59:51.164505 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 47013us
I20250905 19:59:51.116794 30081 raft_consensus.cc:2802] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election won for term 5
W20250905 19:59:51.216686 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:51.169544 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:51.168911 29826 raft_consensus.cc:2747] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:51.053594 30063 raft_consensus.cc:513] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:51.050920 30170 raft_consensus.cc:513] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:51.242511 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:51.142226 30185 raft_consensus.cc:3058] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:51.245893 27975 leader_election.cc:304] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
W20250905 19:59:51.246277 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:51.154991 30189 raft_consensus.cc:738] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:51.162927 30080 raft_consensus.cc:3058] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:51.159921 30196 raft_consensus.cc:695] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 6 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:51.159590 30072 raft_consensus.cc:491] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:51.260105 30148 raft_consensus.cc:491] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:51.275563 28158 raft_consensus.cc:2466] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:51.169718 30172 raft_consensus.cc:2802] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:51.229574 29884 raft_consensus.cc:491] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:51.326692 30210 consensus_queue.cc:1035] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250905 19:59:51.329116 28718 leader_election.cc:304] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 19:59:51.329107 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 330855us
I20250905 19:59:51.159439 30176 consensus_queue.cc:237] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:51.335712 28154 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39586 (request call id 240) took 1090 ms. Trace:
I20250905 19:59:51.267323 29964 raft_consensus.cc:3053] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:59:51.342608 30100 raft_consensus.cc:491] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:51.360545 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:51.359298 28715 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.619s user 0.019s sys 0.047s
W20250905 19:59:51.364992 28715 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.625s user 0.020s sys 0.047s
W20250905 19:59:51.365403 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 625496us
I20250905 19:59:51.327937 30072 raft_consensus.cc:3058] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:51.349385 28154 rpcz_store.cc:276] 0905 19:59:50.245283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:50.245411 (+ 128us) service_pool.cc:225] Handling call
0905 19:59:50.278958 (+ 33547us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:50.378177 (+ 99219us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:50.378216 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:50.378755 (+ 539us) log.cc:844] Serialized 26 byte log entry
0905 19:59:50.526768 (+148013us) raft_consensus.cc:1643] Marking committed up to 4
0905 19:59:50.526834 (+ 66us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:50.526939 (+ 105us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:51.335386 (+808447us) raft_consensus.cc:1683] finished
0905 19:59:51.335418 (+ 32us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:51.335630 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":96458,"raft.queue_time_us":99251,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"thread_start_us":147547,"threads_started":1,"wal-append.queue_time_us":420074}
I20250905 19:59:51.363372 28718 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 19:59:51.380646 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 143037us
I20250905 19:59:51.360162 29964 raft_consensus.cc:738] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
W20250905 19:59:51.394302 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:51.369475 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:51.363196 30079 leader_election.cc:290] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:51.304811 29941 consensus_queue.cc:1035] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.076s
I20250905 19:59:51.410945 27975 leader_election.cc:304] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:51.372010 29924 raft_consensus.cc:2747] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 19:59:51.335377 30148 raft_consensus.cc:513] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:51.375999 30132 raft_consensus.cc:2802] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:51.413113 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:51.426993 29964 consensus_queue.cc:260] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:51.428146 29964 raft_consensus.cc:3058] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:51.429423 30213 consensus_queue.cc:1035] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20250905 19:59:51.430333 30205 log.cc:927] Time spent T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20250905 19:59:51.434576 30148 leader_election.cc:290] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:51.419252 30063 leader_election.cc:290] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:51.423314 28744 raft_consensus.cc:1273] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250905 19:59:51.427613 28750 raft_consensus.cc:3058] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Advancing to term 6
I20250905 19:59:51.416610 30196 consensus_queue.cc:237] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:51.383738 30106 raft_consensus.cc:3053] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:51.437883 29644 raft_consensus.cc:2747] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:51.387166 30209 raft_consensus.cc:2802] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:51.362684 29884 raft_consensus.cc:513] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:51.446817 30209 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:51.447218 30209 raft_consensus.cc:3058] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:51.448369 29884 leader_election.cc:290] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:51.456781 30214 log.cc:927] Time spent T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
W20250905 19:59:51.457341 29864 consensus_meta.cc:287] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.591s user 0.010s sys 0.002s
I20250905 19:59:51.413283 30189 consensus_queue.cc:260] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:51.459651 30201 consensus_queue.cc:1035] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250905 19:59:51.441479 30132 raft_consensus.cc:695] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
W20250905 19:59:51.482604 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 71768.3us
W20250905 19:59:51.460355 30198 log.cc:927] Time spent T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20250905 19:59:51.412233 30220 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:51.374563 30100 raft_consensus.cc:513] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:51.480434 28750 raft_consensus.cc:2466] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 6.
I20250905 19:59:51.439136 29924 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:51.445156 30106 raft_consensus.cc:738] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:51.441332 29905 raft_consensus.cc:3058] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:51.503527 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:51.460688 30170 leader_election.cc:290] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:51.470903 30072 raft_consensus.cc:513] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:51.534554 28004 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.000s sys 0.045s
W20250905 19:59:51.538203 28004 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.519s user 0.000s sys 0.045s
I20250905 19:59:51.504645 30201 raft_consensus.cc:491] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:51.535547 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:51.583142 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:51.542979 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:51.582729 27975 leader_election.cc:304] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:51.479055 29864 raft_consensus.cc:513] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:51.624756 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 93317.5us
I20250905 19:59:51.598620 30148 raft_consensus.cc:2747] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250905 19:59:51.609135 28076 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.011s sys 0.017s
W20250905 19:59:51.641106 30204 log.cc:927] Time spent T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.317s user 0.001s sys 0.000s
I20250905 19:59:51.642741 27975 leader_election.cc:304] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
W20250905 19:59:51.643957 30221 log.cc:927] Time spent T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.135s user 0.000s sys 0.000s
I20250905 19:59:51.647151 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:51.647495 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:51.648212 28081 raft_consensus.cc:2366] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3.
I20250905 19:59:51.650468 28050 leader_election.cc:304] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 19:59:51.655323 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:51.655965 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:51.664002 28083 raft_consensus.cc:2366] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5.
I20250905 19:59:51.664479 28079 raft_consensus.cc:1273] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250905 19:59:51.532343 29924 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:51.525696 30220 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:51.673919 27975 leader_election.cc:304] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:51.675570 27975 leader_election.cc:304] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:51.676216 30213 raft_consensus.cc:491] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:51.677009 27975 leader_election.cc:304] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:51.676856 30213 raft_consensus.cc:513] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:51.678210 30223 raft_consensus.cc:491] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:51.678608 30166 raft_consensus.cc:2802] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:51.679147 30166 raft_consensus.cc:695] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:51.670334 29924 leader_election.cc:290] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:51.678776 30223 raft_consensus.cc:513] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:51.679682 30063 raft_consensus.cc:2747] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250905 19:59:51.679762 29924 raft_consensus.cc:2747] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 19:59:51.681123 30148 raft_consensus.cc:2747] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:51.680035 30166 consensus_queue.cc:237] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:51.666723 30106 consensus_queue.cc:260] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:51.597422 30201 raft_consensus.cc:513] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:51.683465 30106 raft_consensus.cc:3058] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:51.684834 30201 leader_election.cc:290] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:51.686553 30223 leader_election.cc:290] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:51.687548 30072 leader_election.cc:290] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:51.680621 30063 raft_consensus.cc:2747] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 19:59:51.673954 30220 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 19:59:51.647845 28076 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.626s user 0.012s sys 0.017s
I20250905 19:59:51.698303 30063 consensus_queue.cc:1035] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:51.699170 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 19:59:51.700054 30017 consensus_meta.cc:287] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.725s user 0.010s sys 0.000s
I20250905 19:59:51.700484 30017 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:51.701438 30189 raft_consensus.cc:3058] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:51.707837 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:51.708745 30080 raft_consensus.cc:513] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:51.710088 28076 raft_consensus.cc:1273] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250905 19:59:51.710412 28083 raft_consensus.cc:1238] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:51.710991 30080 leader_election.cc:290] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:51.711704 28155 raft_consensus.cc:1238] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:51.711808 28081 raft_consensus.cc:1238] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250905 19:59:51.712257 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:51.712924 28078 raft_consensus.cc:1811] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Handling vote request from an unknown peer 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:51.713081 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 19:59:51.713371 30226 log.cc:927] Time spent T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.106s user 0.000s sys 0.000s
I20250905 19:59:51.713559 28078 raft_consensus.cc:2408] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for term 2 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20250905 19:59:51.713909 28080 raft_consensus.cc:2391] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 5: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:51.714329 28076 raft_consensus.cc:1238] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20250905 19:59:51.707957 29906 consensus_meta.cc:287] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.690s user 0.008s sys 0.003s
W20250905 19:59:51.718910 29995 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.961s user 0.020s sys 0.029s
W20250905 19:59:51.754155 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:51.753894 28158 raft_consensus.cc:1273] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:59:51.740455 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:51.761880 28156 raft_consensus.cc:3058] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:51.759758 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:51.763175 28152 raft_consensus.cc:2466] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
W20250905 19:59:51.739530 29992 consensus_meta.cc:287] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.981s user 0.012s sys 0.000s
W20250905 19:59:51.769371 27975 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:51.770004 28158 raft_consensus.cc:1238] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:51.770078 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:51.766673 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:51.773918 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:51.774016 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:51.775298 28157 raft_consensus.cc:2408] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250905 19:59:51.775434 28152 raft_consensus.cc:2408] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250905 19:59:51.775774 28151 raft_consensus.cc:1238] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 4. Current term is 5. Ops: []
I20250905 19:59:51.778033 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:51.781491 29644 consensus_queue.cc:1035] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:51.783219 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:51.783437 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:51.784008 28078 raft_consensus.cc:2466] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 19:59:51.784247 28077 raft_consensus.cc:2408] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250905 19:59:51.773617 30148 raft_consensus.cc:491] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:51.785399 30148 raft_consensus.cc:513] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:51.786576 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:51.787420 28159 raft_consensus.cc:2408] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250905 19:59:51.787658 30148 leader_election.cc:290] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:51.789511 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:51.789810 30232 raft_consensus.cc:491] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:51.738049 30017 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:51.762769 29924 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:51.645396 30132 consensus_queue.cc:237] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:51.807931 28154 raft_consensus.cc:1238] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 2. Current term is 3. Ops: []
W20250905 19:59:51.765597 29995 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.008s user 0.025s sys 0.030s
I20250905 19:59:51.812489 30100 leader_election.cc:290] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:51.815131 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:51.765794 30170 consensus_queue.cc:1046] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20250905 19:59:51.816648 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:51.824769 28157 raft_consensus.cc:3058] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Advancing to term 7
I20250905 19:59:51.779739 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
W20250905 19:59:51.820148 30225 log.cc:927] Time spent T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20250905 19:59:51.820897 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:51.756076 29906 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:51.829133 28156 raft_consensus.cc:2466] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
I20250905 19:59:51.830482 28744 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Advancing to term 8
I20250905 19:59:51.832152 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:51.822893 30172 raft_consensus.cc:695] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:51.791385 29992 raft_consensus.cc:513] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:51.809285 29864 leader_election.cc:290] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:51.840736 27973 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:51.849794 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:51.853798 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:51.854660 28081 raft_consensus.cc:2391] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 3: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
W20250905 19:59:51.839818 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 474947us
I20250905 19:59:51.862373 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:51.862602 28079 raft_consensus.cc:1238] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:51.830951 30100 raft_consensus.cc:3053] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:59:51.810245 30213 leader_election.cc:290] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:51.862985 28125 leader_election.cc:304] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 19:59:51.873304 28715 leader_election.cc:336] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.774s (SENT)
I20250905 19:59:51.877115 28715 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:51.790390 30232 raft_consensus.cc:513] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:51.890117 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:51.890833 28009 raft_consensus.cc:3058] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
W20250905 19:59:51.863657 30081 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.021s sys 0.029s
I20250905 19:59:51.888367 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:51.893465 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:51.893792 28156 raft_consensus.cc:2391] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 4: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:51.853071 30063 raft_consensus.cc:491] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:51.877655 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:51.911377 28080 raft_consensus.cc:1238] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:51.895362 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:51.893334 30081 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.625s user 0.022s sys 0.029s
I20250905 19:59:51.913369 30081 raft_consensus.cc:695] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:51.913409 28081 raft_consensus.cc:2466] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 2.
I20250905 19:59:51.907179 28077 raft_consensus.cc:1273] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250905 19:59:51.914557 30081 consensus_queue.cc:237] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:51.915354 30235 log.cc:927] Time spent T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
W20250905 19:59:51.915974 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:51.917104 30185 consensus_meta.cc:287] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.651s user 0.003s sys 0.008s
I20250905 19:59:51.917518 30185 raft_consensus.cc:513] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
W20250905 19:59:51.919313 30228 log.cc:927] Time spent T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20250905 19:59:51.921916 30063 raft_consensus.cc:513] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:51.909860 29905 raft_consensus.cc:2747] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:51.924510 30063 leader_election.cc:290] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:51.927806 28154 raft_consensus.cc:2466] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 19:59:51.931002 28157 raft_consensus.cc:2466] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 7.
I20250905 19:59:51.874619 30100 raft_consensus.cc:738] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:51.827173 29924 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: NON_VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: true } }
I20250905 19:59:51.949512 27973 leader_election.cc:304] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:51.932826 30232 leader_election.cc:290] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:51.957896 28009 raft_consensus.cc:2466] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 3.
I20250905 19:59:51.888293 30140 raft_consensus.cc:2747] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250905 19:59:51.963627 28715 leader_election.cc:336] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.865s (SENT)
I20250905 19:59:51.961436 28125 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:51.983981 28715 leader_election.cc:304] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:51.982312 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 19:59:51.945849 30225 log.cc:927] Time spent T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.072s user 0.001s sys 0.000s
I20250905 19:59:51.985147 28718 leader_election.cc:304] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:51.985958 28081 raft_consensus.cc:1238] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250905 19:59:51.894867 30213 raft_consensus.cc:2802] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 19:59:51.960548 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:51.988215 30234 consensus_queue.cc:1035] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.007s
I20250905 19:59:51.988600 30238 raft_consensus.cc:2747] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250905 19:59:51.989598 29924 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:51.990044 30148 raft_consensus.cc:2802] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250905 19:59:51.981297 30209 consensus_meta.cc:287] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.516s user 0.010s sys 0.002s
I20250905 19:59:51.997766 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:51.960732 29941 raft_consensus.cc:491] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:51.981591 30202 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.602s user 0.018s sys 0.030s
W20250905 19:59:51.981510 30165 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.730s user 0.023s sys 0.047s
I20250905 19:59:52.021081 28125 leader_election.cc:304] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:52.024597 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:52.037578 28080 raft_consensus.cc:1273] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250905 19:59:52.021510 30213 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250905 19:59:52.042085 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:52.045761 28125 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:52.023407 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:52.043390 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:52.023586 29840 raft_consensus.cc:2802] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader election won for term 3
W20250905 19:59:52.037525 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 234052us
I20250905 19:59:52.043159 28157 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:59:51.960639 30063 consensus_queue.cc:1046] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.073s
I20250905 19:59:51.948879 30172 consensus_queue.cc:237] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:52.049829 28156 raft_consensus.cc:2466] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 19:59:52.029747 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:52.059511 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:52.050745 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:52.028292 30148 raft_consensus.cc:491] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:52.051002 28125 leader_election.cc:304] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 19:59:52.051600 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:52.057858 28155 raft_consensus.cc:1273] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20250905 19:59:52.068305 28715 leader_election.cc:336] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.841s (SENT)
I20250905 19:59:52.064965 28080 raft_consensus.cc:1273] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250905 19:59:52.088354 28154 raft_consensus.cc:2466] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
W20250905 19:59:52.033797 29943 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.590s user 0.009s sys 0.013s
I20250905 19:59:52.089939 28009 raft_consensus.cc:3058] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
W20250905 19:59:52.047189 30165 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.796s user 0.023s sys 0.048s
W20250905 19:59:52.047161 30202 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.674s user 0.019s sys 0.036s
I20250905 19:59:52.088323 28081 raft_consensus.cc:2408] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 5 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20250905 19:59:52.098320 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:52.094123 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:52.146243 28154 raft_consensus.cc:2466] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 19:59:52.106050 28152 raft_consensus.cc:1238] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: []
W20250905 19:59:52.118592 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:52.048508 30213 raft_consensus.cc:3058] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
I20250905 19:59:52.148629 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:52.150285 28050 leader_election.cc:304] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 19:59:52.132329 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:52.050086 30238 raft_consensus.cc:2802] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:52.028177 30240 consensus_queue.cc:1035] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.065s
I20250905 19:59:52.151767 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:52.034053 30209 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:52.172050 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:52.155788 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:52.055325 28076 raft_consensus.cc:2466] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:59:52.089013 30148 raft_consensus.cc:3058] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:52.155696 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:52.177536 28007 raft_consensus.cc:2466] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
W20250905 19:59:52.103593 29943 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.660s user 0.010s sys 0.014s
W20250905 19:59:52.153560 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:52.058084 29840 raft_consensus.cc:695] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:52.182341 28156 raft_consensus.cc:3058] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:52.201184 28009 raft_consensus.cc:2466] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 19:59:52.183038 30242 consensus_queue.cc:1035] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.088s
I20250905 19:59:52.183156 30201 consensus_queue.cc:1046] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:52.183810 29905 raft_consensus.cc:491] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:52.185170 29884 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:52.185874 28751 raft_consensus.cc:1273] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250905 19:59:52.185711 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:52.230722 28002 raft_consensus.cc:3058] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:52.037750 30100 consensus_queue.cc:260] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:52.183174 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:52.229117 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:52.189821 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:52.232348 28081 raft_consensus.cc:3058] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:52.187053 29992 leader_election.cc:290] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:52.190142 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:52.256973 28010 raft_consensus.cc:2466] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:52.257926 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:52.139467 30246 raft_consensus.cc:2802] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 19:59:52.258649 30246 raft_consensus.cc:491] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:52.213845 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:52.259472 28007 raft_consensus.cc:3058] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:52.152709 30106 consensus_queue.cc:1035] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.067s
W20250905 19:59:52.198866 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 48634.4us
I20250905 19:59:52.277208 27973 leader_election.cc:304] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:52.231823 28008 raft_consensus.cc:2466] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:52.191267 30185 leader_election.cc:290] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:52.259044 30246 raft_consensus.cc:3058] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:52.210971 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:52.235777 30233 raft_consensus.cc:2802] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader election won for term 7
I20250905 19:59:52.235877 30100 raft_consensus.cc:3058] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
W20250905 19:59:52.236739 30236 log.cc:927] Time spent T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
W20250905 19:59:52.237533 30250 log.cc:927] Time spent T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
W20250905 19:59:52.238262 30245 log.cc:927] Time spent T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.095s user 0.000s sys 0.001s
I20250905 19:59:52.238476 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:52.238694 29848 raft_consensus.cc:491] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:52.239488 30239 log.cc:927] Time spent T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.199s user 0.001s sys 0.000s
I20250905 19:59:52.239768 29884 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:52.230067 30234 raft_consensus.cc:491] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:52.199651 30243 raft_consensus.cc:491] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:52.232568 29905 raft_consensus.cc:513] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:52.188835 30238 raft_consensus.cc:491] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:52.350570 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 19:59:52.242825 30227 log.cc:927] Time spent T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.345s user 0.000s sys 0.001s
I20250905 19:59:52.242357 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:52.245560 28084 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1415) took 1410 ms. Trace:
I20250905 19:59:52.246976 29840 consensus_queue.cc:237] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:52.246999 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:52.250308 28159 raft_consensus.cc:1273] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20250905 19:59:52.250362 30189 consensus_meta.cc:287] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.548s user 0.007s sys 0.004s
I20250905 19:59:52.250819 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:52.252346 30232 consensus_queue.cc:1035] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20250905 19:59:52.253912 28157 raft_consensus.cc:1273] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20250905 19:59:52.384369 28157 pending_rounds.cc:85] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting all ops after (but not including) 5
I20250905 19:59:52.384682 28157 pending_rounds.cc:107] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20250905 19:59:52.384953 28157 raft_consensus.cc:2887] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250905 19:59:52.190768 30155 raft_consensus.cc:2747] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 19:59:52.255214 29906 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:52.256875 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:52.391651 28752 raft_consensus.cc:2391] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 4: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
W20250905 19:59:52.387502 30166 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.700s user 0.013s sys 0.025s
I20250905 19:59:52.261884 28718 leader_election.cc:304] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:52.392560 29905 leader_election.cc:290] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 19:59:52.392804 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 131033us
I20250905 19:59:52.393996 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:52.394888 28082 raft_consensus.cc:2466] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 19:59:52.048981 29941 raft_consensus.cc:513] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:52.395978 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:52.397629 29941 leader_election.cc:290] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:52.216962 30196 raft_consensus.cc:2802] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election won for term 5
I20250905 19:59:52.360512 30243 raft_consensus.cc:513] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:52.405373 30196 raft_consensus.cc:695] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
W20250905 19:59:52.234951 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:52.408697 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:52.408793 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:52.409381 28078 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Advancing to term 8
W20250905 19:59:52.307272 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:52.307792 28156 raft_consensus.cc:2466] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:52.412984 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:52.308174 28007 raft_consensus.cc:2466] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 19:59:52.309080 28009 raft_consensus.cc:1215] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[2.2-2.2] Dedup: 2.2->[]
I20250905 19:59:52.317056 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:52.318332 27975 leader_election.cc:304] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:52.331881 28081 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 19:59:52.337639 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:52.347159 28001 raft_consensus.cc:2391] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 4: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 19:59:52.311084 30080 raft_consensus.cc:491] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:52.361877 28002 raft_consensus.cc:2466] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 19:59:52.343751 28077 raft_consensus.cc:3058] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:52.381701 28154 raft_consensus.cc:3058] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:52.343839 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:52.364912 28084 rpcz_store.cc:276] 0905 19:59:50.835589 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:50.844785 (+ 9196us) service_pool.cc:225] Handling call
0905 19:59:50.855569 (+ 10784us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:50.863236 (+ 7667us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:50.863273 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:50.878623 (+ 15350us) log.cc:844] Serialized 26 byte log entry
0905 19:59:51.482246 (+603623us) spinlock_profiling.cc:243] Waited 28.7 ms on lock 0x7b180047b540. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49ac86d47 00007fe4a38dcb99 00007fe49aca369b 00007fe49aca193f 00007fe49aca3c02 00007fe49d8572bb 00007fe49d85cdc3 00007fe49d85d171 00007fe49dbec09f
0905 19:59:51.983347 (+501101us) raft_consensus.cc:1643] Marking committed up to 6
0905 19:59:52.016546 (+ 33199us) log.cc:844] Serialized 14 byte log entry
0905 19:59:52.019142 (+ 2596us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:52.019249 (+ 107us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:52.245154 (+225905us) raft_consensus.cc:1683] finished
0905 19:59:52.245270 (+ 116us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:52.245452 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5282,"prepare.queue_time_us":37438,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":12177,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":11547,"spinlock_wait_cycles":70794240,"thread_start_us":499419,"threads_started":1,"wal-append.queue_time_us":1308355}
I20250905 19:59:52.389417 28153 raft_consensus.cc:2408] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250905 19:59:52.345269 30233 raft_consensus.cc:695] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
W20250905 19:59:52.392257 30166 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.707s user 0.013s sys 0.026s
I20250905 19:59:52.400461 28082 raft_consensus.cc:2466] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
W20250905 19:59:52.400012 27973 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:52.403067 30132 raft_consensus.cc:2802] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader election won for term 5
I20250905 19:59:52.404551 28004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 245) took 1450 ms. Trace:
I20250905 19:59:52.364003 30238 raft_consensus.cc:3058] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:52.406255 30196 consensus_queue.cc:237] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:52.357991 29848 raft_consensus.cc:513] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:52.457269 30189 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:52.971122 28009 raft_consensus.cc:1238] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250905 19:59:52.971470 30243 leader_election.cc:290] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:52.591384 28078 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 8.
I20250905 19:59:52.972376 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:59:52.973104 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 654919us
I20250905 19:59:52.973203 28009 raft_consensus.cc:2466] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
W20250905 19:59:52.653498 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:52.946588 28079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1370) took 1278 ms. Trace:
I20250905 19:59:52.972705 28078 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1422) took 1094 ms. Trace:
I20250905 19:59:52.973759 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:52.978919 28001 raft_consensus.cc:2391] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:52.980473 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:52.490701 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 19:59:52.681872 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 137918us
I20250905 19:59:52.464016 28005 raft_consensus.cc:1238] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:52.980963 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:52.983804 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:52.984031 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:52.984622 28005 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 19:59:52.981408 28007 raft_consensus.cc:2366] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 2. Current term is 3.
I20250905 19:59:52.524641 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:59:52.617534 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:52.982051 28156 raft_consensus.cc:3058] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
W20250905 19:59:52.587024 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:52.979475 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:52.975498 28079 rpcz_store.cc:276] 0905 19:59:51.668075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:51.704707 (+ 36632us) service_pool.cc:225] Handling call
0905 19:59:51.708331 (+ 3624us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:51.863063 (+154732us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:52.946195 (+1083132us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x7b540029b654. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:52.946495 (+ 300us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":354097920}
W20250905 19:59:52.734768 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 469834us
W20250905 19:59:52.986960 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:52.423774 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:52.552438 28080 raft_consensus.cc:2466] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 19:59:52.987321 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:52.423005 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:52.992264 28003 raft_consensus.cc:2408] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20250905 19:59:52.985283 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 504847us
I20250905 19:59:52.974239 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:52.991453 28007 raft_consensus.cc:2391] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
W20250905 19:59:52.817444 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:52.993307 28009 raft_consensus.cc:2391] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 5: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 19:59:52.971529 30189 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:52.977037 28078 rpcz_store.cc:276] 0905 19:59:51.878345 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:51.884544 (+ 6199us) service_pool.cc:225] Handling call
0905 19:59:52.408630 (+524086us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b5800204f98. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b7db60 00007fe4a2b7d444 00007fe49c2050c4 00007fe49c205052
0905 19:59:52.972651 (+564021us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":25863424}
I20250905 19:59:52.997185 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:52.998162 28002 raft_consensus.cc:2466] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
I20250905 19:59:52.996785 28125 leader_election.cc:304] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:52.977813 28004 rpcz_store.cc:276] 0905 19:59:50.954554 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:50.954727 (+ 173us) service_pool.cc:225] Handling call
0905 19:59:50.955320 (+ 593us) raft_consensus.cc:1469] Updating replica for 2 ops
0905 19:59:50.968675 (+ 13355us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:50.968723 (+ 48us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0905 19:59:50.992380 (+ 23657us) log.cc:844] Serialized 52 byte log entry
0905 19:59:51.539920 (+547540us) raft_consensus.cc:1643] Marking committed up to 6
0905 19:59:51.603017 (+ 63097us) log.cc:844] Serialized 14 byte log entry
0905 19:59:51.603703 (+ 686us) log.cc:844] Serialized 14 byte log entry
0905 19:59:51.603962 (+ 259us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:51.604060 (+ 98us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:52.404323 (+800263us) raft_consensus.cc:1683] finished
0905 19:59:52.404343 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:52.404490 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":65129,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":82,"raft.queue_time_us":11547,"raft.run_cpu_time_us":358,"raft.run_wall_time_us":18670,"thread_start_us":521078,"threads_started":1,"wal-append.queue_time_us":812175}
I20250905 19:59:53.000335 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:53.000697 27975 leader_election.cc:304] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:53.002677 27975 leader_election.cc:304] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:53.003537 27975 leader_election.cc:304] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:53.004536 28001 raft_consensus.cc:2366] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4.
I20250905 19:59:52.989718 28159 raft_consensus.cc:2466] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:53.007059 27975 leader_election.cc:304] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
W20250905 19:59:52.526189 30257 log.cc:927] Time spent T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.126s user 0.000s sys 0.000s
W20250905 19:59:52.650503 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:53.020802 28156 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 19:59:53.009810 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:53.021973 28125 leader_election.cc:304] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:53.010946 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:59:52.527125 30259 log.cc:927] Time spent T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20250905 19:59:53.016377 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:53.014310 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:52.479624 30185 raft_consensus.cc:491] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:53.013360 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:52.734526 30017 raft_consensus.cc:2802] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:53.023515 28009 raft_consensus.cc:2466] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:53.023566 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:53.023877 28154 raft_consensus.cc:2466] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 19:59:53.044626 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:52.997159 30240 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:53.045835 28154 raft_consensus.cc:1273] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250905 19:59:52.563653 30201 raft_consensus.cc:491] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:52.869308 29941 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:52.464417 30165 raft_consensus.cc:491] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:53.051862 29941 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:52.691926 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:53.028184 28085 raft_consensus.cc:1238] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:52.877029 30100 raft_consensus.cc:491] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:52.940533 29884 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 19:59:53.029022 30148 consensus_meta.cc:287] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.817s user 0.006s sys 0.002s
I20250905 19:59:53.028920 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:53.060242 28004 raft_consensus.cc:2391] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 5: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:53.029789 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:53.031029 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:53.061168 28078 raft_consensus.cc:2391] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 5: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:53.031095 28156 raft_consensus.cc:1238] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:53.031390 30189 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:52.848140 30170 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:52.828934 30106 raft_consensus.cc:491] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:53.034482 30247 raft_consensus.cc:2802] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250905 19:59:53.004745 28744 consensus_meta.cc:287] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 1.173s user 0.012s sys 0.001s
I20250905 19:59:53.037520 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:52.801719 30261 log.cc:927] Time spent T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
I20250905 19:59:53.101879 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:53.038415 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:53.039242 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:53.039261 28007 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
I20250905 19:59:52.703925 30232 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:53.040120 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:53.103783 28081 raft_consensus.cc:2466] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 19:59:53.069432 29884 raft_consensus.cc:2747] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:52.976410 30243 raft_consensus.cc:2747] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:52.703692 30263 raft_consensus.cc:491] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:53.035398 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:53.005576 29905 raft_consensus.cc:2747] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:52.528786 30209 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:52.996100 30072 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:53.006146 29906 raft_consensus.cc:2747] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:52.360689 30234 raft_consensus.cc:513] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:53.107750 30247 raft_consensus.cc:491] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:53.051370 28003 raft_consensus.cc:2391] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 4: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:53.051952 30165 raft_consensus.cc:513] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:53.026221 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:53.068284 28010 raft_consensus.cc:1273] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250905 19:59:52.996793 30080 raft_consensus.cc:513] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:53.038595 30166 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:53.227553 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:53.106853 28007 raft_consensus.cc:1273] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:59:53.079792 28083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1377) took 1368 ms. Trace:
I20250905 19:59:53.112603 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:53.112751 28008 raft_consensus.cc:2466] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
I20250905 19:59:53.113724 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:53.116601 28004 raft_consensus.cc:1273] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:59:53.229988 28084 raft_consensus.cc:2366] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3.
I20250905 19:59:53.072835 28009 raft_consensus.cc:1238] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:53.199086 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:53.208159 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:53.233433 28009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1940) took 1001 ms. Trace:
I20250905 19:59:53.233884 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:53.234265 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:53.235178 28082 raft_consensus.cc:2466] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 3.
I20250905 19:59:53.235420 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:53.236388 28081 raft_consensus.cc:2366] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a for earlier term 7. Current term is 8.
I20250905 19:59:53.236249 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:53.237293 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:53.059718 30201 raft_consensus.cc:513] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:53.009436 29924 raft_consensus.cc:2747] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 19:59:53.066475 30148 raft_consensus.cc:513] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:53.049860 30185 raft_consensus.cc:513] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:53.054544 30240 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:53.243038 30148 leader_election.cc:290] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:53.219955 28010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1348) took 1071 ms. Trace:
W20250905 19:59:53.224416 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 213518us
I20250905 19:59:53.243942 28010 rpcz_store.cc:276] 0905 19:59:52.148814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:53.037262 (+888448us) service_pool.cc:225] Handling call
0905 19:59:53.055466 (+ 18204us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:53.219588 (+164122us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:53.219878 (+ 290us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:53.223661 28156 raft_consensus.cc:1273] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250905 19:59:53.122287 30267 raft_consensus.cc:2802] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:53.074152 30170 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:53.065708 30100 raft_consensus.cc:513] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:53.228343 28083 rpcz_store.cc:276] 0905 19:59:51.711344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:51.724241 (+ 12897us) service_pool.cc:225] Handling call
0905 19:59:51.739132 (+ 14891us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:51.767202 (+ 28070us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:51.767248 (+ 46us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:51.791978 (+ 24730us) log.cc:844] Serialized 26 byte log entry
0905 19:59:52.044586 (+252608us) raft_consensus.cc:1643] Marking committed up to 1
0905 19:59:52.044686 (+ 100us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:52.044891 (+ 205us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:53.079568 (+1034677us) raft_consensus.cc:1683] finished
0905 19:59:53.079588 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:53.079729 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":215862,"threads_started":1,"wal-append.queue_time_us":518571}
W20250905 19:59:53.229904 28157 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.510s user 0.002s sys 0.005s
I20250905 19:59:53.119215 28005 raft_consensus.cc:2408] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20250905 19:59:53.102283 30106 raft_consensus.cc:513] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:53.117028 30232 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:53.167493 30263 raft_consensus.cc:513] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:53.237617 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:53.121908 30233 consensus_queue.cc:237] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:53.233796 28009 rpcz_store.cc:276] 0905 19:59:52.237732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:53.072147 (+834415us) service_pool.cc:225] Handling call
0905 19:59:53.072592 (+ 445us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:53.233205 (+160613us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:53.233382 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:53.336407 28008 raft_consensus.cc:1273] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:59:53.252844 30267 raft_consensus.cc:491] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:53.550290 28002 raft_consensus.cc:2466] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 7.
W20250905 19:59:53.337369 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:53.565230 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 19:59:53.540335 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 509875us
I20250905 19:59:53.368548 28076 raft_consensus.cc:2408] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250905 19:59:53.221274 30247 raft_consensus.cc:3058] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:53.375304 28079 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
I20250905 19:59:53.587610 28079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1394) took 1341 ms. Trace:
I20250905 19:59:53.588482 28079 rpcz_store.cc:276] 0905 19:59:52.246641 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:53.005374 (+758733us) service_pool.cc:225] Handling call
0905 19:59:53.587512 (+582138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:53.221084 29826 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:53.279829 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:53.263849 28003 raft_consensus.cc:1238] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:53.590260 28078 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 19:59:53.590618 28003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1947) took 1350 ms. Trace:
I20250905 19:59:53.413127 28007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1942) took 1174 ms. Trace:
I20250905 19:59:53.591513 28007 rpcz_store.cc:276] 0905 19:59:52.238567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:53.103924 (+865357us) service_pool.cc:225] Handling call
0905 19:59:53.104257 (+ 333us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:53.228209 (+123952us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:53.412819 (+184610us) spinlock_profiling.cc:243] Waited 2.25 ms on lock 0x7b5400054bd4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:53.413060 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5169920}
W20250905 19:59:53.541916 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 534847us
I20250905 19:59:53.339587 28750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40918 (request call id 501) took 1429 ms. Trace:
I20250905 19:59:53.549113 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:53.594741 27975 leader_election.cc:304] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a
W20250905 19:59:53.444049 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:53.591230 28003 rpcz_store.cc:276] 0905 19:59:52.240197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:53.228820 (+988623us) service_pool.cc:225] Handling call
0905 19:59:53.263564 (+ 34744us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:53.590202 (+326638us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:53.590541 (+ 339us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:53.596513 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:53.549592 28005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1948) took 1309 ms. Trace:
I20250905 19:59:53.596859 28155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2016) took 1494 ms. Trace:
I20250905 19:59:53.597160 28005 rpcz_store.cc:276] 0905 19:59:52.240475 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:52.983531 (+743056us) service_pool.cc:225] Handling call
0905 19:59:53.549524 (+565993us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:53.542454 29941 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:53.211094 30072 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:53.518730 28154 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2022) took 1107 ms. Trace:
I20250905 19:59:53.553059 28082 raft_consensus.cc:2391] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 3: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:53.550961 28008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1956) took 1300 ms. Trace:
I20250905 19:59:53.554023 28010 raft_consensus.cc:1238] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20250905 19:59:53.230273 30246 consensus_meta.cc:287] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.893s user 0.012s sys 0.000s
W20250905 19:59:53.376987 28715 leader_election.cc:336] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.683s (SENT)
I20250905 19:59:53.605654 29826 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:53.608947 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 232015us
W20250905 19:59:53.253259 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 195617us
I20250905 19:59:53.610184 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:53.610699 29826 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:53.247251 30166 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:53.611749 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:53.363293 30132 raft_consensus.cc:695] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:53.565151 28157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39586 (request call id 253) took 1315 ms. Trace:
I20250905 19:59:53.613104 30166 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:53.394241 28152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39586 (request call id 246) took 1146 ms. Trace:
I20250905 19:59:53.580332 30185 leader_election.cc:290] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:53.566817 28158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39586 (request call id 250) took 1317 ms. Trace:
I20250905 19:59:53.613627 30132 consensus_queue.cc:237] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:53.614593 28158 rpcz_store.cc:276] 0905 19:59:52.249218 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:52.252261 (+ 3043us) service_pool.cc:225] Handling call
0905 19:59:52.252515 (+ 254us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:53.248232 (+995717us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:53.248280 (+ 48us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:53.248340 (+ 60us) raft_consensus.cc:1643] Marking committed up to 6
0905 19:59:53.248392 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:53.566554 (+318162us) spinlock_profiling.cc:243] Waited 995 ms on lock 0x7b540009ddd4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:53.566610 (+ 56us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:53.566756 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2289334656}
I20250905 19:59:53.363386 29848 leader_election.cc:290] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:53.615957 30072 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:53.602027 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:53.550967 30267 raft_consensus.cc:3058] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:53.618115 28010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1965) took 1361 ms. Trace:
I20250905 19:59:53.618597 28010 rpcz_store.cc:276] 0905 19:59:52.298388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:53.252090 (+953702us) service_pool.cc:225] Handling call
0905 19:59:53.548507 (+296417us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:53.617787 (+ 69280us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:53.618039 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:53.619976 30263 leader_election.cc:290] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:53.606637 28008 rpcz_store.cc:276] 0905 19:59:52.250281 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:53.229551 (+979270us) service_pool.cc:225] Handling call
0905 19:59:53.267082 (+ 37531us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:53.550586 (+283504us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:53.550796 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:53.620760 30100 leader_election.cc:290] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:53.621701 28008 raft_consensus.cc:1273] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250905 19:59:53.621912 30201 leader_election.cc:290] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:53.543175 28004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1964) took 1287 ms. Trace:
I20250905 19:59:53.582355 28083 raft_consensus.cc:2391] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 3: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:53.577989 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:53.393241 30213 consensus_meta.cc:287] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.175s user 0.010s sys 0.000s
I20250905 19:59:53.394773 30240 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 19:59:53.549827 28122 leader_election.cc:336] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 2.785s (SENT)
W20250905 19:59:53.582682 28125 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.000s sys 0.007s
I20250905 19:59:53.541764 30284 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:53.394279 30170 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:53.593678 28750 rpcz_store.cc:276] 0905 19:59:51.910679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:51.960358 (+ 49679us) service_pool.cc:225] Handling call
0905 19:59:51.982224 (+ 21866us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:51.982476 (+ 252us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:51.982501 (+ 25us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:51.982976 (+ 475us) log.cc:844] Serialized 26 byte log entry
0905 19:59:52.186476 (+203500us) raft_consensus.cc:1643] Marking committed up to 6
0905 19:59:52.186515 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:52.186587 (+ 72us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:53.339307 (+1152720us) raft_consensus.cc:1683] finished
0905 19:59:53.339343 (+ 36us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:53.339514 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":143443,"threads_started":1,"wal-append.queue_time_us":434846}
I20250905 19:59:53.603255 28007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 254) took 1327 ms. Trace:
W20250905 19:59:53.541595 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:53.603847 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:53.597247 28155 rpcz_store.cc:276] 0905 19:59:52.102313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:52.102581 (+ 268us) service_pool.cc:225] Handling call
0905 19:59:52.156609 (+ 54028us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:52.157643 (+ 1034us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:52.157691 (+ 48us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:52.199475 (+ 41784us) log.cc:844] Serialized 26 byte log entry
0905 19:59:52.425842 (+226367us) raft_consensus.cc:1643] Marking committed up to 2
0905 19:59:52.425902 (+ 60us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:52.426000 (+ 98us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:53.596599 (+1170599us) raft_consensus.cc:1683] finished
0905 19:59:53.596622 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:53.596801 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":389,"raft.queue_time_us":1134,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":10318,"thread_start_us":188903,"threads_started":1,"wal-append.queue_time_us":1113819}
I20250905 19:59:53.119649 28744 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 8.
I20250905 19:59:53.605260 27975 leader_election.cc:304] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:53.607303 28005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 256) took 1331 ms. Trace:
I20250905 19:59:53.606256 28154 rpcz_store.cc:276] 0905 19:59:52.411691 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:53.090667 (+678976us) service_pool.cc:225] Handling call
0905 19:59:53.117951 (+ 27284us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:53.248459 (+130508us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:53.248518 (+ 59us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:53.248599 (+ 81us) raft_consensus.cc:1643] Marking committed up to 3
0905 19:59:53.248684 (+ 85us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:53.518353 (+269669us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b54000ec9d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:53.518451 (+ 98us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:53.518647 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":299102080}
I20250905 19:59:53.608775 29906 consensus_queue.cc:1035] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250905 19:59:53.269368 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 233986us
I20250905 19:59:53.622498 28004 rpcz_store.cc:276] 0905 19:59:52.256144 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:53.230639 (+974495us) service_pool.cc:225] Handling call
0905 19:59:53.542295 (+311656us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:53.542710 (+ 415us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:53.542754 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:53.542816 (+ 62us) raft_consensus.cc:1643] Marking committed up to 1
0905 19:59:53.542868 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:53.542948 (+ 80us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:53.543102 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13251,"raft.queue_time_us":9236,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":38159}
I20250905 19:59:53.618023 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:53.976943 28125 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.936s user 0.000s sys 0.028s
I20250905 19:59:53.977522 28078 raft_consensus.cc:2391] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 4: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:53.978391 28751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40904 (request call id 545) took 1575 ms. Trace:
I20250905 19:59:53.978637 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:53.978878 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:53.979281 28081 raft_consensus.cc:2466] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 19:59:53.979360 30148 raft_consensus.cc:2802] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:53.684597 29964 raft_consensus.cc:491] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:53.979879 30148 raft_consensus.cc:491] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250905 19:59:53.980049 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 374838us
I20250905 19:59:53.980412 30148 raft_consensus.cc:3058] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:53.980935 27975 leader_election.cc:304] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:53.981846 30283 raft_consensus.cc:2747] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:53.981592 30284 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:53.983194 27975 leader_election.cc:304] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:53.983953 30284 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 19:59:53.984217 28744 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40904 (request call id 538) took 2248 ms (client timeout 2019 ms). Trace:
I20250905 19:59:53.742611 30232 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:53.984635 28744 rpcz_store.cc:269] 0905 19:59:51.735913 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:51.737095 (+ 1182us) service_pool.cc:225] Handling call
0905 19:59:53.984119 (+2247024us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:53.661535 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:53.986161 28084 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 19:59:53.915952 28050 leader_election.cc:304] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:53.987304 29826 raft_consensus.cc:2802] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:53.766696 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 19:59:53.818496 28715 leader_election.cc:336] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.787s (SENT)
I20250905 19:59:53.987869 29826 raft_consensus.cc:695] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
W20250905 19:59:53.988598 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 170173us
I20250905 19:59:53.749192 28002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1945) took 1509 ms. Trace:
I20250905 19:59:53.988838 29826 consensus_queue.cc:237] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:53.989403 28715 leader_election.cc:336] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 2.058s (SENT)
W20250905 19:59:53.989881 28080 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1390) took 1809 ms (client timeout 1604 ms). Trace:
I20250905 19:59:53.989929 28715 leader_election.cc:304] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:53.987156 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.010561 28009 raft_consensus.cc:2953] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:53.700047 28082 raft_consensus.cc:2466] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6.
I20250905 19:59:53.986953 29964 raft_consensus.cc:513] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:53.980968 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:54.013445 28159 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.678s user 0.009s sys 0.033s
W20250905 19:59:54.072240 28159 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.737s user 0.013s sys 0.035s
W20250905 19:59:54.072351 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 82979us
I20250905 19:59:53.613821 28152 rpcz_store.cc:276] 0905 19:59:52.247528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:52.252726 (+ 5198us) service_pool.cc:225] Handling call
0905 19:59:52.253018 (+ 292us) raft_consensus.cc:1469] Updating replica for 2 ops
0905 19:59:52.292491 (+ 39473us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:52.292525 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0905 19:59:52.317633 (+ 25108us) log.cc:844] Serialized 52 byte log entry
0905 19:59:52.720268 (+402635us) raft_consensus.cc:1643] Marking committed up to 5
0905 19:59:52.720326 (+ 58us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:52.720427 (+ 101us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:53.394027 (+673600us) raft_consensus.cc:1683] finished
0905 19:59:53.394050 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:53.394180 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9371,"raft.queue_time_us":85070,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":10604,"thread_start_us":329143,"threads_started":1,"wal-append.queue_time_us":850444}
W20250905 19:59:53.990408 28080 rpcz_store.cc:269] 0905 19:59:52.180971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:52.222206 (+ 41235us) service_pool.cc:225] Handling call
0905 19:59:53.989688 (+1767482us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b540036d654. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:53.989807 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":382994176}
W20250905 19:59:54.036676 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:53.608290 30246 raft_consensus.cc:513] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:53.694093 30185 raft_consensus.cc:491] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:53.977162 28007 rpcz_store.cc:276] 0905 19:59:52.275703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:53.602060 (+1326357us) service_pool.cc:225] Handling call
0905 19:59:53.602530 (+ 470us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:53.602791 (+ 261us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:53.602832 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:53.602893 (+ 61us) raft_consensus.cc:1643] Marking committed up to 5
0905 19:59:53.602945 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:53.603021 (+ 76us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:53.603179 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:53.713469 30166 raft_consensus.cc:491] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:54.081522 30166 raft_consensus.cc:513] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:53.990744 28005 rpcz_store.cc:276] 0905 19:59:52.276299 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:53.606484 (+1330185us) service_pool.cc:225] Handling call
0905 19:59:53.606786 (+ 302us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:53.606974 (+ 188us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:53.607008 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:53.607053 (+ 45us) raft_consensus.cc:1643] Marking committed up to 5
0905 19:59:53.607084 (+ 31us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:53.607140 (+ 56us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:53.607249 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:53.994187 30283 raft_consensus.cc:2802] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 19:59:53.978888 28751 rpcz_store.cc:276] 0905 19:59:52.402869 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:52.406577 (+ 3708us) service_pool.cc:225] Handling call
0905 19:59:52.406950 (+ 373us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:52.407316 (+ 366us) raft_consensus.cc:1532] Early marking committed up to index 5
0905 19:59:52.985460 (+578144us) log.cc:844] Serialized 14 byte log entry
0905 19:59:53.277237 (+291777us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:53.335924 (+ 58687us) log.cc:844] Serialized 26 byte log entry
0905 19:59:53.336098 (+ 174us) raft_consensus.cc:1643] Marking committed up to 5
0905 19:59:53.336143 (+ 45us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:53.336237 (+ 94us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:53.773514 (+437277us) raft_consensus.cc:1683] finished
0905 19:59:53.773542 (+ 28us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:53.978334 (+204792us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":587666,"prepare.queue_time_us":710551,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":6247,"raft.queue_time_us":20422,"raft.run_cpu_time_us":295,"raft.run_wall_time_us":119370,"thread_start_us":675735,"threads_started":2,"wal-append.queue_time_us":756295}
I20250905 19:59:53.619993 28010 raft_consensus.cc:1238] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:54.084658 28010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 257) took 1808 ms. Trace:
I20250905 19:59:54.085057 28010 rpcz_store.cc:276] 0905 19:59:52.276575 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:53.619477 (+1342902us) service_pool.cc:225] Handling call
0905 19:59:53.619827 (+ 350us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:54.084313 (+464486us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:54.084563 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:53.758832 28718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.702s user 0.011s sys 0.018s
I20250905 19:59:53.991235 28085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1375) took 2280 ms. Trace:
W20250905 19:59:54.086696 28718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.030s user 0.016s sys 0.033s
I20250905 19:59:53.716593 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 19:59:54.036689 28076 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1391) took 1808 ms (client timeout 2197 ms). Trace:
I20250905 19:59:54.026672 28078 raft_consensus.cc:2408] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250905 19:59:53.993287 27973 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 19:59:54.106114 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 112907us
I20250905 19:59:54.077157 30093 raft_consensus.cc:2747] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 19:59:53.999037 29906 raft_consensus.cc:491] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
W20250905 19:59:53.993248 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 77439.9us
I20250905 19:59:53.608378 29924 raft_consensus.cc:2802] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250905 19:59:54.055438 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:54.116042 28153 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33752 (request call id 1463) took 2347 ms (client timeout 1594 ms). Trace:
W20250905 19:59:53.703256 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 90446.4us
I20250905 19:59:53.683597 30280 raft_consensus.cc:491] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
W20250905 19:59:53.750205 28077 consensus_meta.cc:287] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.736s user 0.008s sys 0.000s
I20250905 19:59:53.713847 30287 raft_consensus.cc:491] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:53.608381 30165 leader_election.cc:290] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:54.037320 28008 raft_consensus.cc:1273] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250905 19:59:53.786020 30017 raft_consensus.cc:491] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:53.746205 30176 raft_consensus.cc:3053] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:54.043557 28004 raft_consensus.cc:1273] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250905 19:59:54.161368 28004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 260) took 1162 ms. Trace:
I20250905 19:59:54.161753 28004 rpcz_store.cc:276] 0905 19:59:52.998680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:53.976791 (+978111us) service_pool.cc:225] Handling call
0905 19:59:54.043086 (+ 66295us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:54.161066 (+117980us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:54.161310 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:53.989214 28002 rpcz_store.cc:276] 0905 19:59:52.239579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:53.010059 (+770480us) service_pool.cc:225] Handling call
0905 19:59:53.748912 (+738853us) spinlock_profiling.cc:243] Waited 512 ms on lock 0x7b54000ff5d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:53.749096 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1177062656}
I20250905 19:59:53.999217 30213 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:53.772718 30241 log.cc:927] Time spent T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 1.588s user 0.002s sys 0.011s
W20250905 19:59:53.977422 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 955475us
I20250905 19:59:53.613205 28157 rpcz_store.cc:276] 0905 19:59:52.250043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:52.394336 (+144293us) service_pool.cc:225] Handling call
0905 19:59:52.394662 (+ 326us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:52.394859 (+ 197us) raft_consensus.cc:1532] Early marking committed up to index 5
0905 19:59:52.719795 (+324936us) log.cc:844] Serialized 14 byte log entry
0905 19:59:53.564728 (+844933us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:53.564799 (+ 71us) raft_consensus.cc:1643] Marking committed up to 5
0905 19:59:53.564862 (+ 63us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:53.564936 (+ 74us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:53.565079 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15478,"prepare.queue_time_us":825877,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":58608,"thread_start_us":1153227,"threads_started":3}
I20250905 19:59:54.086658 30185 raft_consensus.cc:513] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:53.683425 30269 log.cc:927] Time spent T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.480s user 0.000s sys 0.001s
I20250905 19:59:54.072068 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:53.766155 29995 raft_consensus.cc:3053] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:54.005401 27975 leader_election.cc:304] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:54.074364 28084 raft_consensus.cc:2391] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 4: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:54.075517 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.207010 28079 raft_consensus.cc:2466] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 3.
W20250905 19:59:54.032083 30037 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.011s sys 0.034s
I20250905 19:59:53.977988 30234 leader_election.cc:290] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 19:59:54.092361 28715 leader_election.cc:336] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 2.100s (SENT)
I20250905 19:59:54.086889 28085 rpcz_store.cc:276] 0905 19:59:51.710631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:51.712289 (+ 1658us) service_pool.cc:225] Handling call
0905 19:59:51.712615 (+ 326us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:53.058574 (+1345959us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:53.991000 (+932426us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b540004e2d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:53.991178 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3007267200}
I20250905 19:59:53.682312 28003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 255) took 1406 ms. Trace:
I20250905 19:59:54.224349 28715 leader_election.cc:304] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:54.103387 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.104346 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:54.083142 30278 raft_consensus.cc:491] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:54.083585 30106 leader_election.cc:290] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:54.107203 30292 raft_consensus.cc:491] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:54.086405 30283 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
W20250905 19:59:54.010564 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 460726us
W20250905 19:59:54.116489 28153 rpcz_store.cc:269] 0905 19:59:51.759545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:51.759672 (+ 127us) service_pool.cc:225] Handling call
0905 19:59:54.097713 (+2338041us) spinlock_profiling.cc:243] Waited 610 ms on lock 0x7b54002d0854. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:54.107003 (+ 9290us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1403259264}
W20250905 19:59:54.104213 28051 connection.cc:232] RPC call timeout handler was delayed by -1.0284547609999208s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16570000000000001s for a response
I20250905 19:59:54.119287 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:54.230862 28078 raft_consensus.cc:2374] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6. Re-sending same reply.
W20250905 19:59:54.120924 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:54.123008 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:54.072875 30148 raft_consensus.cc:513] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:54.110126 30263 raft_consensus.cc:2802] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader election won for term 8
I20250905 19:59:54.110345 30100 raft_consensus.cc:491] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:54.117247 29941 raft_consensus.cc:491] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:54.122743 28077 raft_consensus.cc:2466] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 19:59:54.124547 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:54.124853 30166 leader_election.cc:290] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:54.118764 29906 raft_consensus.cc:513] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:54.119066 30132 raft_consensus.cc:491] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:54.126793 30246 leader_election.cc:290] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:54.128141 30201 raft_consensus.cc:491] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:54.125077 28050 leader_election.cc:304] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:54.116251 29924 raft_consensus.cc:491] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
W20250905 19:59:54.105455 28076 rpcz_store.cc:269] 0905 19:59:52.228470 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:52.230105 (+ 1635us) service_pool.cc:225] Handling call
0905 19:59:53.976075 (+1745970us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b54001f09d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:54.032056 (+ 55981us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":576864128}
W20250905 19:59:54.143520 30268 log.cc:927] Time spent T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.667s user 0.010s sys 0.000s
I20250905 19:59:54.110415 30290 raft_consensus.cc:491] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:54.144039 30080 leader_election.cc:290] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 19:59:54.144614 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:54.115773 29964 leader_election.cc:290] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:54.156019 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:54.157974 28008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 259) took 1182 ms. Trace:
I20250905 19:59:53.703220 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.168125 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:54.166298 28125 leader_election.cc:304] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:54.171887 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 19:59:54.175629 28717 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.635s user 0.053s sys 0.040s
I20250905 19:59:54.160786 30017 raft_consensus.cc:3058] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:54.187392 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 19:59:54.173454 30247 consensus_meta.cc:287] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.568s user 0.010s sys 0.000s
I20250905 19:59:54.196599 30072 consensus_queue.cc:1035] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250905 19:59:54.196801 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:54.198606 28009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1356) took 1923 ms. Trace:
I20250905 19:59:54.334141 28009 rpcz_store.cc:276] 0905 19:59:52.275269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:53.561143 (+1285874us) service_pool.cc:225] Handling call
0905 19:59:53.561753 (+ 610us) raft_consensus.cc:1469] Updating replica for 2 ops
0905 19:59:53.596402 (+ 34649us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:53.596437 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0905 19:59:53.715193 (+118756us) log.cc:844] Serialized 590 byte log entry
0905 19:59:54.010383 (+295190us) raft_consensus.cc:1643] Marking committed up to 4
0905 19:59:54.129421 (+119038us) log.cc:844] Serialized 14 byte log entry
0905 19:59:54.198182 (+ 68761us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:54.198268 (+ 86us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:54.198317 (+ 49us) raft_consensus.cc:1683] finished
0905 19:59:54.198335 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:54.198529 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":75059,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":5976,"thread_start_us":214704,"threads_started":1,"wal-append.queue_time_us":395324}
I20250905 19:59:54.201846 30093 consensus_queue.cc:1046] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250905 19:59:54.203384 30299 consensus_queue.cc:1035] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20250905 19:59:54.160929 30176 raft_consensus.cc:738] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
W20250905 19:59:54.195045 30291 log.cc:927] Time spent T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.314s user 0.000s sys 0.000s
W20250905 19:59:54.187855 30238 consensus_meta.cc:287] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.175s user 0.012s sys 0.002s
I20250905 19:59:54.207512 28006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1345) took 2059 ms. Trace:
W20250905 19:59:54.209230 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 203937us
I20250905 19:59:54.224772 28003 rpcz_store.cc:276] 0905 19:59:52.275987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:53.603592 (+1327605us) service_pool.cc:225] Handling call
0905 19:59:53.681661 (+ 78069us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:53.681881 (+ 220us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:53.681919 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:53.681978 (+ 59us) raft_consensus.cc:1643] Marking committed up to 7
0905 19:59:53.682017 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:53.682086 (+ 69us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:53.682238 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:54.174765 30185 leader_election.cc:290] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:54.175092 29995 raft_consensus.cc:738] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:54.341871 28006 rpcz_store.cc:276] 0905 19:59:52.147976 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:52.151122 (+ 3146us) service_pool.cc:225] Handling call
0905 19:59:52.184008 (+ 32886us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:52.234703 (+ 50695us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:52.234775 (+ 72us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:52.253258 (+ 18483us) log.cc:844] Serialized 26 byte log entry
0905 19:59:52.720716 (+467458us) raft_consensus.cc:1643] Marking committed up to 1
0905 19:59:52.720767 (+ 51us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:52.720858 (+ 91us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:54.207176 (+1486318us) raft_consensus.cc:1683] finished
0905 19:59:54.207211 (+ 35us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:54.207424 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":929,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"thread_start_us":508971,"threads_started":2,"wal-append.queue_time_us":812255}
I20250905 19:59:54.343261 27975 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:54.343392 29995 consensus_queue.cc:260] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:54.343720 30176 consensus_queue.cc:260] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:54.344192 29884 raft_consensus.cc:2802] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:54.344764 29995 raft_consensus.cc:3058] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:54.344908 30176 raft_consensus.cc:3058] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
W20250905 19:59:54.248811 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:54.346624 30238 raft_consensus.cc:513] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:54.347980 30232 consensus_queue.cc:1035] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250905 19:59:54.348366 28151 raft_consensus.cc:1238] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:54.250069 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.350363 28084 raft_consensus.cc:2408] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250905 19:59:54.351336 28151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2020) took 2245 ms. Trace:
I20250905 19:59:54.351406 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.233870 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.351828 28151 rpcz_store.cc:276] 0905 19:59:52.105502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:52.105624 (+ 122us) service_pool.cc:225] Handling call
0905 19:59:52.149296 (+ 43672us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:54.348732 (+2199436us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:54.351098 (+ 2366us) spinlock_profiling.cc:243] Waited 2.2 s on lock 0x7b5400073fd4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:54.351258 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5056914176}
I20250905 19:59:54.352921 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:54.353416 30307 log.cc:927] Time spent T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20250905 19:59:54.296494 30290 raft_consensus.cc:513] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:54.353777 28085 raft_consensus.cc:2466] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 1.
I20250905 19:59:54.255947 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:54.223938 30037 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.721s user 0.011s sys 0.035s
I20250905 19:59:54.260517 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.162711 30280 raft_consensus.cc:513] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:54.259856 30100 raft_consensus.cc:513] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:54.267103 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 149254us
I20250905 19:59:54.224329 30262 raft_consensus.cc:2747] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:54.269165 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:54.161489 30287 raft_consensus.cc:513] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:54.271221 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 146215us
I20250905 19:59:54.272217 30166 raft_consensus.cc:2802] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250905 19:59:54.272715 28051 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:59564 dropped due to backpressure: service queue is full with 50 items
I20250905 19:59:54.233824 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:54.233294 30283 raft_consensus.cc:3058] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 19:59:54.257826 30234 raft_consensus.cc:491] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:54.264365 30263 raft_consensus.cc:695] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:54.278972 30132 raft_consensus.cc:513] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:54.297623 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:54.287319 29924 raft_consensus.cc:3058] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 19:59:54.407052 30100 leader_election.cc:290] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:54.406690 27908 catalog_manager.cc:5582] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:54.287256 30201 raft_consensus.cc:513] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:54.407944 30304 log.cc:927] Time spent T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.113s user 0.000s sys 0.001s
I20250905 19:59:54.408959 30189 consensus_queue.cc:1035] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250905 19:59:54.409353 30201 leader_election.cc:290] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:54.408768 27908 catalog_manager.cc:5582] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 1 to 5, leader changed from e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 5 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:54.410032 28082 raft_consensus.cc:2466] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
W20250905 19:59:54.411093 30308 log.cc:927] Time spent T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.118s user 0.001s sys 0.000s
I20250905 19:59:54.411818 30280 leader_election.cc:290] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:54.414058 30283 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:54.415228 30287 leader_election.cc:290] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:54.416270 29826 consensus_queue.cc:1035] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.012s
W20250905 19:59:54.417069 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 71732.3us
W20250905 19:59:54.286942 28077 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1389) took 2170 ms (client timeout 2016 ms). Trace:
W20250905 19:59:54.303412 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 100034us
I20250905 19:59:54.254084 30292 raft_consensus.cc:513] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:54.313143 28008 rpcz_store.cc:276] 0905 19:59:52.975671 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:53.976116 (+1000445us) service_pool.cc:225] Handling call
0905 19:59:54.032785 (+ 56669us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:54.157565 (+124780us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:54.157887 (+ 322us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:54.314354 27907 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:60818 (request call id 147) took 4715 ms. Trace:
I20250905 19:59:54.421672 30292 leader_election.cc:290] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:54.314848 28717 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.773s user 0.061s sys 0.049s
W20250905 19:59:54.422539 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 881819us
W20250905 19:59:54.423041 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:54.424165 30148 leader_election.cc:290] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:54.317766 28083 raft_consensus.cc:2374] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6. Re-sending same reply.
I20250905 19:59:54.426417 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 19:59:54.320571 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 154395us
I20250905 19:59:54.404194 30106 consensus_queue.cc:1035] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20250905 19:59:54.426838 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:54.427356 28125 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:54.428135 30106 raft_consensus.cc:2747] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 19:59:54.321312 30247 raft_consensus.cc:513] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:54.428557 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:54.429869 28748 raft_consensus.cc:2466] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:54.430083 30247 leader_election.cc:290] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:54.245711 30278 raft_consensus.cc:513] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 19:59:54.336078 30233 raft_consensus.cc:2802] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250905 19:59:54.340759 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 248451us
W20250905 19:59:54.431547 28715 leader_election.cc:336] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 2.119s (SENT)
I20250905 19:59:54.363622 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:54.432349 28078 raft_consensus.cc:3058] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:54.432024 30263 consensus_queue.cc:237] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:54.419282 28077 rpcz_store.cc:269] 0905 19:59:52.096509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:52.139106 (+ 42597us) service_pool.cc:225] Handling call
0905 19:59:54.266735 (+2127629us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:54.433563 28079 raft_consensus.cc:2466] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 19:59:54.374013 30289 raft_consensus.cc:3053] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:54.263897 29941 raft_consensus.cc:513] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:54.256726 30279 raft_consensus.cc:491] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:54.417398 28080 raft_consensus.cc:2391] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 3: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:54.417841 27975 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:54.405072 30234 raft_consensus.cc:513] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:54.371243 29964 consensus_queue.cc:1035] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.066s
W20250905 19:59:54.455302 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:54.425428 28744 raft_consensus.cc:2408] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250905 19:59:54.430925 30233 raft_consensus.cc:491] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:54.372047 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.468345 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:54.469204 28154 raft_consensus.cc:2408] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250905 19:59:54.428573 28125 leader_election.cc:304] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:54.470065 28156 pending_rounds.cc:85] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting all ops after (but not including) 1
I20250905 19:59:54.470368 28156 pending_rounds.cc:107] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250905 19:59:54.470322 28125 leader_election.cc:304] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
W20250905 19:59:54.380558 30297 log.cc:927] Time spent T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.081s user 0.002s sys 0.000s
I20250905 19:59:54.437228 28079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1405) took 1466 ms. Trace:
I20250905 19:59:54.470767 28156 raft_consensus.cc:2887] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250905 19:59:54.485248 28079 rpcz_store.cc:276] 0905 19:59:52.971256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:54.223811 (+1252555us) service_pool.cc:225] Handling call
0905 19:59:54.436999 (+213188us) spinlock_profiling.cc:243] Waited 78.8 ms on lock 0x7b54003aa554. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:54.437150 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":181284736}
I20250905 19:59:54.468679 28123 leader_election.cc:304] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
W20250905 19:59:54.487087 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:54.440400 28078 raft_consensus.cc:2466] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 19:59:54.418282 27908 catalog_manager.cc:5582] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 2 to 5, leader changed from 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) to 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132). New cstate: current_term: 5 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:54.488552 27908 catalog_manager.cc:5582] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 4 to 5, leader changed from e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 5 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } }
I20250905 19:59:54.442081 28122 leader_election.cc:400] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Vote denied by peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193) with higher term. Message: Invalid argument: T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3.
I20250905 19:59:54.377506 30166 raft_consensus.cc:491] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:54.373216 29884 raft_consensus.cc:695] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:54.370939 30313 raft_consensus.cc:2747] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:54.381948 29906 leader_election.cc:290] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:54.444777 28001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1350) took 2295 ms. Trace:
I20250905 19:59:54.399338 30213 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:54.447751 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:54.449203 28158 raft_consensus.cc:1238] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:54.373354 28051 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:59564 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1419)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50660 (request call id 240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50660 (request call id 241)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50660 (request call id 225)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1423)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1426)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50660 (request call id 205)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1421)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50660 (request call id 238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50660 (request call id 201)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50660 (request call id 239)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1424)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1418)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1428)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1433)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50660 (request call id 229)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50660 (request call id 230)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1420)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1396)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50660 (request call id 233)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1422)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1431)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1432)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1430)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1429)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:51630 (request call id 12)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1392)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1393)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1397)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1430)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1431)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1398)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1399)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1406)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 197)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 198)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1434)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 203)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 218)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 221)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 222)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 223)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 232)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 236)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 242)
I20250905 19:59:54.403612 30317 raft_consensus.cc:491] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:54.371423 30185 consensus_queue.cc:1035] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.065s
I20250905 19:59:54.453765 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:59:54.462916 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 45166.9us
I20250905 19:59:54.510635 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:54.460816 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.511868 28083 raft_consensus.cc:2466] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 1.
I20250905 19:59:54.420068 30283 leader_election.cc:290] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:54.501134 28001 rpcz_store.cc:276] 0905 19:59:52.149301 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:53.040693 (+891392us) service_pool.cc:225] Handling call
0905 19:59:53.046949 (+ 6256us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:53.469905 (+422956us) spinlock_profiling.cc:243] Waited 7.49 ms on lock 0x7b180037b7c0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49d86555c 00007fe49ac873cb 00007fe4a38dcb99 00007fe49aca369b 00007fe49aca193f 00007fe49aca1455 00007fe49dc11020 00007fe49dc11bbb 00007fe49dc1c5fb
0905 19:59:53.470157 (+ 252us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:53.470196 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:53.542199 (+ 72003us) log.cc:844] Serialized 26 byte log entry
0905 19:59:54.090905 (+548706us) raft_consensus.cc:1643] Marking committed up to 1
0905 19:59:54.090960 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:54.091058 (+ 98us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:54.417172 (+326114us) raft_consensus.cc:1683] finished
0905 19:59:54.417191 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:54.444512 (+ 27321us) spinlock_profiling.cc:243] Waited 5.83 ms on lock 0x7b5400290c50. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:54.444697 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":158337,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":8627,"spinlock_wait_cycles":30648960,"thread_start_us":659427,"threads_started":2,"wal-append.queue_time_us":645775}
I20250905 19:59:54.467769 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:54.513361 28157 raft_consensus.cc:2466] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 6.
I20250905 19:59:54.435097 29924 raft_consensus.cc:513] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:54.518532 28152 raft_consensus.cc:1273] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250905 19:59:54.518750 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.468060 28150 raft_consensus.cc:1238] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:54.519713 28001 raft_consensus.cc:1273] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250905 19:59:54.472045 28125 leader_election.cc:304] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:54.472615 28008 raft_consensus.cc:1273] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20250905 19:59:54.475288 30267 consensus_meta.cc:287] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.858s user 0.010s sys 0.005s
I20250905 19:59:54.475174 30262 raft_consensus.cc:2747] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 19:59:54.475709 30080 raft_consensus.cc:2802] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:54.476478 28010 raft_consensus.cc:1238] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:54.476835 29941 leader_election.cc:290] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:54.476929 30106 raft_consensus.cc:2747] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:54.477366 30233 raft_consensus.cc:3058] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:54.478026 30290 leader_election.cc:290] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:54.442898 30289 raft_consensus.cc:738] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:54.473212 30072 consensus_queue.cc:1035] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20250905 19:59:54.478875 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:54.463752 30287 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:54.451782 30196 consensus_queue.cc:1046] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20250905 19:59:54.541589 30196 raft_consensus.cc:3053] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:59:54.542001 30196 raft_consensus.cc:738] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:54.542656 29884 consensus_queue.cc:237] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:54.542922 30196 consensus_queue.cc:260] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:54.544106 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:54.544270 30196 raft_consensus.cc:3058] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:54.479749 28085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1432) took 1456 ms. Trace:
I20250905 19:59:54.546128 28085 rpcz_store.cc:276] 0905 19:59:53.022889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:54.371778 (+1348889us) service_pool.cc:225] Handling call
0905 19:59:54.479654 (+107876us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:54.547268 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.547539 30232 raft_consensus.cc:491] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:54.548470 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.549369 28085 raft_consensus.cc:2391] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:54.549705 28085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50660 (request call id 205) took 1393 ms. Trace:
I20250905 19:59:54.549904 28085 rpcz_store.cc:276] 0905 19:59:53.156117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:54.548226 (+1392109us) service_pool.cc:225] Handling call
0905 19:59:54.549661 (+ 1435us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:54.480679 30132 leader_election.cc:290] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:54.482439 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:54.387387 30246 consensus_queue.cc:1046] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.066s
I20250905 19:59:54.484027 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:54.554085 28154 raft_consensus.cc:2408] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250905 19:59:54.484223 30234 leader_election.cc:290] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:54.465073 30238 leader_election.cc:290] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 19:59:54.432413 28715 leader_election.cc:336] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 2.263s (SENT)
I20250905 19:59:54.490111 28122 leader_election.cc:403] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250905 19:59:54.487725 28078 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50660 (request call id 212) took 1115 ms. Trace:
I20250905 19:59:54.559342 28078 rpcz_store.cc:276] 0905 19:59:53.372167 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:54.355715 (+983548us) service_pool.cc:225] Handling call
0905 19:59:54.487666 (+131951us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:54.559454 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 127052us
I20250905 19:59:54.491552 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:54.560647 28002 raft_consensus.cc:2466] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 3.
W20250905 19:59:54.492060 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:54.494778 28080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50660 (request call id 202) took 1446 ms. Trace:
W20250905 19:59:54.561517 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:54.495769 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:54.561579 28080 rpcz_store.cc:276] 0905 19:59:53.050724 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:54.079259 (+1028535us) service_pool.cc:225] Handling call
0905 19:59:54.494563 (+415304us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b540029ac54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:54.494723 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":441634560}
I20250905 19:59:54.562356 28079 raft_consensus.cc:3058] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:54.492305 27908 catalog_manager.cc:5582] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } }
I20250905 19:59:54.465476 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:54.504261 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.568236 28082 raft_consensus.cc:2408] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for term 7 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250905 19:59:54.493702 30166 raft_consensus.cc:3058] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:54.466921 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:54.420815 27907 rpcz_store.cc:276] 0905 19:59:49.591451 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:49.591630 (+ 179us) service_pool.cc:225] Handling call
0905 19:59:54.306724 (+4715094us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:59:50.454660 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:59:50.481206 (+ 26546us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:59:50.481283 (+ 77us) write_op.cc:435] Acquired schema lock
0905 19:59:50.481311 (+ 28us) tablet.cc:660] Decoding operations
0905 19:59:50.482557 (+ 1246us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:59:50.482671 (+ 114us) write_op.cc:641] Partition lock acquired for write op
0905 19:59:50.482717 (+ 46us) tablet.cc:683] Acquiring locks for 6 operations
0905 19:59:50.483670 (+ 953us) tablet.cc:699] Row locks acquired
0905 19:59:50.483700 (+ 30us) write_op.cc:260] PREPARE: finished
0905 19:59:50.483956 (+ 256us) write_op.cc:270] Start()
0905 19:59:50.484186 (+ 230us) write_op.cc:276] Timestamp: P: 1757102390483854 usec, L: 0
0905 19:59:50.484221 (+ 35us) op_driver.cc:348] REPLICATION: starting
0905 19:59:50.485137 (+ 916us) log.cc:844] Serialized 2452 byte log entry
0905 19:59:52.386297 (+1901160us) op_driver.cc:464] REPLICATION: finished
0905 19:59:53.204185 (+817888us) write_op.cc:301] APPLY: starting
0905 19:59:53.217338 (+ 13153us) tablet.cc:1365] starting BulkCheckPresence
0905 19:59:53.280938 (+ 63600us) tablet.cc:1368] finished BulkCheckPresence
0905 19:59:53.363103 (+ 82165us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:59:53.828292 (+465189us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:59:53.828598 (+ 306us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0905 19:59:53.828669 (+ 71us) write_op.cc:312] APPLY: finished
0905 19:59:53.829036 (+ 367us) log.cc:844] Serialized 53 byte log entry
0905 19:59:54.167558 (+338522us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:59:54.167697 (+ 139us) write_op.cc:454] Released schema lock
0905 19:59:54.167750 (+ 53us) write_op.cc:341] FINISH: Updating metrics
0905 19:59:54.203532 (+ 35782us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:59:54.203564 (+ 32us) write_op.cc:454] Released schema lock
0905 19:59:54.352858 (+149294us) spinlock_profiling.cc:243] Waited 1.27 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49d85891e 00007fe49d857c66 00007fe49d857888 00007fe49d8612c1 00007fe49d861279 00007fe49d861209
Metrics: {"reported_tablets":14,"child_traces":[["op",{"apply.queue_time_us":717066,"apply.run_cpu_time_us":50498,"apply.run_wall_time_us":1082378,"num_ops":6,"prepare.queue_time_us":407322,"prepare.run_cpu_time_us":47722,"prepare.run_wall_time_us":390756,"raft.queue_time_us":939993,"raft.run_cpu_time_us":9255,"raft.run_wall_time_us":469723,"replication_time_us":1861806,"spinlock_wait_cycles":2929408,"thread_start_us":1944630,"threads_started":5,"wal-append.queue_time_us":931867,"wal-append.run_cpu_time_us":66684,"wal-append.run_wall_time_us":2209799}]]}
I20250905 19:59:54.538079 30262 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:54.569384 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:54.514412 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:54.464205 30279 raft_consensus.cc:513] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:54.493649 30280 raft_consensus.cc:2802] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:54.493770 30189 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:54.519390 28083 raft_consensus.cc:2466] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
W20250905 19:59:54.524537 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 52560.1us
I20250905 19:59:54.532780 30037 consensus_queue.cc:1035] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
W20250905 19:59:54.526928 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 64895.5us
I20250905 19:59:54.529095 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:54.532028 28050 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:54.577976 28150 raft_consensus.cc:2466] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
W20250905 19:59:54.578150 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 158491us
I20250905 19:59:54.534090 28007 raft_consensus.cc:1238] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 5. Current term is 6. Ops: []
I20250905 19:59:54.527489 30213 raft_consensus.cc:2802] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:54.535449 30299 consensus_queue.cc:1046] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20250905 19:59:54.539564 28153 raft_consensus.cc:2391] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 6: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:54.527478 30317 raft_consensus.cc:513] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:54.540372 30287 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:54.525803 30165 raft_consensus.cc:491] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:54.432073 30278 leader_election.cc:290] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:54.562870 28159 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1480) took 2153 ms. Trace:
I20250905 19:59:54.565104 28718 leader_election.cc:304] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:54.562042 28715 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:54.565354 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:54.534855 30080 raft_consensus.cc:695] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:54.570729 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.564098 27908 catalog_manager.cc:5582] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 1 to 3, leader changed from e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 3 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } }
I20250905 19:59:54.570322 28158 raft_consensus.cc:2466] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
I20250905 19:59:54.548252 30232 raft_consensus.cc:513] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:54.575582 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.526743 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:54.551170 30319 raft_consensus.cc:491] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:54.578351 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 46466us
I20250905 19:59:54.580996 28122 leader_election.cc:304] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:54.535800 29941 consensus_queue.cc:1046] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20250905 19:59:54.606858 28050 leader_election.cc:304] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:54.526631 30267 raft_consensus.cc:513] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:54.564208 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.530829 30314 consensus_queue.cc:1035] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.042s
I20250905 19:59:54.566388 30246 consensus_queue.cc:1230] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer e4413ae5e0b64b52a6f360ac580239b9 log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20250905 19:59:54.633411 28081 raft_consensus.cc:2391] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 4: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:54.596765 30072 raft_consensus.cc:2802] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 19:59:54.587951 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 19:59:54.588379 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 48168.3us
I20250905 19:59:54.589720 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.588651 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:54.590955 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:54.586913 28159 rpcz_store.cc:276] 0905 19:59:52.409357 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:53.017422 (+608065us) service_pool.cc:225] Handling call
0905 19:59:53.032000 (+ 14578us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:53.218340 (+186340us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:53.218391 (+ 51us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:53.218980 (+ 589us) log.cc:844] Serialized 26 byte log entry
0905 19:59:54.072824 (+853844us) raft_consensus.cc:1643] Marking committed up to 3
0905 19:59:54.290606 (+217782us) log.cc:844] Serialized 14 byte log entry
0905 19:59:54.290871 (+ 265us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:54.290975 (+ 104us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:54.346061 (+ 55086us) spinlock_profiling.cc:243] Waited 8.24 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49d85891e 00007fe49d857c66 00007fe49d857888 00007fe49d8612c1 00007fe49d861279 00007fe49d861209
0905 19:59:54.539615 (+193554us) raft_consensus.cc:1683] finished
0905 19:59:54.562559 (+ 22944us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:54.562786 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14657,"prepare.queue_time_us":261718,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":189283,"raft.run_cpu_time_us":320,"raft.run_wall_time_us":7697,"spinlock_wait_cycles":18942208,"thread_start_us":1123061,"threads_started":3,"wal-append.queue_time_us":983147}
I20250905 19:59:54.595003 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:54.574704 28748 raft_consensus.cc:2374] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 8. Re-sending same reply.
I20250905 19:59:54.595563 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:54.647251 29941 consensus_queue.cc:1230] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer 26b2f874b8e54f08b01a9ed00d5cce05 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250905 19:59:54.582716 30189 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:54.589679 28080 raft_consensus.cc:2374] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5. Re-sending same reply.
I20250905 19:59:54.596349 30093 raft_consensus.cc:2802] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:54.597512 28083 raft_consensus.cc:2408] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250905 19:59:54.597476 27908 catalog_manager.cc:5582] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:54.587581 28085 raft_consensus.cc:2374] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7. Re-sending same reply.
I20250905 19:59:54.650399 27908 catalog_manager.cc:5582] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 4 to 5, leader changed from 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 5 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 19:59:54.651293 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.651705 30189 leader_election.cc:290] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:54.652271 27908 catalog_manager.cc:5582] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 2 to 3, leader changed from e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 3 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } }
I20250905 19:59:54.653079 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.574676 30280 raft_consensus.cc:491] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:54.654304 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.654598 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:54.654240 28085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1433) took 1621 ms. Trace:
I20250905 19:59:54.600494 28079 raft_consensus.cc:2466] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:59:54.655478 30093 raft_consensus.cc:491] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:54.655915 30093 raft_consensus.cc:3058] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:54.656095 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:54.656713 28079 raft_consensus.cc:3058] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 3
I20250905 19:59:54.601235 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:54.576622 30240 raft_consensus.cc:2747] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193) with higher term. Message: Invalid argument: T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3.
I20250905 19:59:54.552217 30316 raft_consensus.cc:491] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:54.611562 28152 raft_consensus.cc:1273] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250905 19:59:54.659049 30316 raft_consensus.cc:513] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:54.581959 30201 raft_consensus.cc:2747] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 19:59:54.583794 30320 raft_consensus.cc:491] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:54.661266 30316 leader_election.cc:290] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:54.661389 30320 raft_consensus.cc:513] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:54.662724 28080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50660 (request call id 201) took 1619 ms. Trace:
I20250905 19:59:54.663065 28080 rpcz_store.cc:276] 0905 19:59:53.043651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:54.570466 (+1526815us) service_pool.cc:225] Handling call
0905 19:59:54.662663 (+ 92197us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:54.663722 30320 leader_election.cc:290] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:54.623113 28122 leader_election.cc:304] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:54.663954 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.664731 28080 raft_consensus.cc:2391] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:54.665128 28080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50660 (request call id 230) took 1043 ms. Trace:
I20250905 19:59:54.665324 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:54.666575 28122 leader_election.cc:304] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
W20250905 19:59:54.627218 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 51064.5us
I20250905 19:59:54.665410 28080 rpcz_store.cc:276] 0905 19:59:53.621226 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:54.663690 (+1042464us) service_pool.cc:225] Handling call
0905 19:59:54.665073 (+ 1383us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:54.655016 28083 raft_consensus.cc:2466] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:59:54.669670 28145 tablet_service.cc:1515] Processing DeleteTablet for tablet 986c4f475a8d46f39465775db70416d3 with delete_type TABLET_DATA_TOMBSTONED (TS 26b2f874b8e54f08b01a9ed00d5cce05 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33742
I20250905 19:59:54.669073 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:54.672876 28080 raft_consensus.cc:2391] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 5: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:54.669972 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.674654 28154 raft_consensus.cc:3058] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
I20250905 19:59:54.675570 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.676416 28080 raft_consensus.cc:2374] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5. Re-sending same reply.
I20250905 19:59:54.677074 28080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50660 (request call id 233) took 1055 ms. Trace:
I20250905 19:59:54.677467 28080 rpcz_store.cc:276] 0905 19:59:53.622121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:54.675209 (+1053088us) service_pool.cc:225] Handling call
0905 19:59:54.677002 (+ 1793us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:54.678527 30280 raft_consensus.cc:3058] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:54.682893 30240 raft_consensus.cc:2747] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250905 19:59:54.683075 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.684023 28080 raft_consensus.cc:2374] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7. Re-sending same reply.
I20250905 19:59:54.655489 28085 rpcz_store.cc:276] 0905 19:59:53.032571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:54.652705 (+1620134us) service_pool.cc:225] Handling call
0905 19:59:54.654166 (+ 1461us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:54.685468 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.686935 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.688033 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:54.688573 28085 raft_consensus.cc:3058] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:54.688714 30326 consensus_queue.cc:1035] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20250905 19:59:54.686573 30240 raft_consensus.cc:2747] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:54.622922 30292 raft_consensus.cc:2802] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 19:59:54.584995 30238 consensus_queue.cc:1035] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.042s
I20250905 19:59:54.699697 28050 leader_election.cc:304] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 19:59:54.700400 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 65158.8us
W20250905 19:59:54.713191 28715 leader_election.cc:336] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 2.542s (SENT)
I20250905 19:59:54.674750 28083 raft_consensus.cc:2466] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 19:59:54.738111 28083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1396) took 2490 ms. Trace:
I20250905 19:59:54.738873 28083 rpcz_store.cc:276] 0905 19:59:52.247201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:54.669715 (+2422514us) service_pool.cc:225] Handling call
0905 19:59:54.738029 (+ 68314us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:54.721266 27908 catalog_manager.cc:5582] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 1 to 2, leader changed from cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) to 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132). New cstate: current_term: 2 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:54.740092 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.741060 28083 raft_consensus.cc:2466] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 7.
I20250905 19:59:54.740382 27908 catalog_manager.cc:5582] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 4 to 7, leader changed from 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 7 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250905 19:59:54.745697 28079 raft_consensus.cc:2466] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 3.
I20250905 19:59:54.746312 28079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50660 (request call id 229) took 1125 ms. Trace:
I20250905 19:59:54.746642 28079 rpcz_store.cc:276] 0905 19:59:53.620916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:54.655839 (+1034923us) service_pool.cc:225] Handling call
0905 19:59:54.746254 (+ 90415us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:54.743546 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:54.755395 28154 raft_consensus.cc:2466] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
I20250905 19:59:54.685397 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.764709 27908 catalog_manager.cc:5582] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } }
I20250905 19:59:54.763046 28080 raft_consensus.cc:2466] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 19:59:54.772171 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.772842 28083 raft_consensus.cc:1811] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Handling vote request from an unknown peer 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:54.773378 28083 raft_consensus.cc:2366] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3.
I20250905 19:59:54.773489 28080 raft_consensus.cc:1273] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:59:54.775261 28083 raft_consensus.cc:1273] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20250905 19:59:54.781411 28007 raft_consensus.cc:1273] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250905 19:59:54.787169 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:54.788002 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.598845 30165 raft_consensus.cc:513] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:54.790609 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:54.793131 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 19:59:54.794376 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 127865us
I20250905 19:59:54.787365 28080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1392) took 2545 ms. Trace:
I20250905 19:59:54.795538 28080 rpcz_store.cc:276] 0905 19:59:52.231499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:54.772057 (+2540558us) service_pool.cc:225] Handling call
0905 19:59:54.772523 (+ 466us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:54.776126 (+ 3603us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:54.776327 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:54.575299 30262 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:54.732208 30292 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:54.700228 30246 raft_consensus.cc:2747] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 19:59:54.645174 30072 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:54.797248 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.776311 28083 pending_rounds.cc:85] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9: Aborting all ops after (but not including) 2
I20250905 19:59:54.803325 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.734855 30247 raft_consensus.cc:491] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
W20250905 19:59:54.786577 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:54.805271 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:54.808108 28083 pending_rounds.cc:107] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20250905 19:59:54.822011 28083 raft_consensus.cc:2887] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250905 19:59:54.820169 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:54.823375 28004 raft_consensus.cc:2466] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:54.828591 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:54.830801 28078 raft_consensus.cc:2466] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:59:54.786701 30037 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:54.833570 28051 leader_election.cc:304] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:54.821502 28006 raft_consensus.cc:2466] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:54.836568 28081 raft_consensus.cc:1273] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250905 19:59:54.614115 30319 raft_consensus.cc:513] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:54.809491 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:54.791850 28077 raft_consensus.cc:2391] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 3: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:54.644142 30299 consensus_queue.cc:1046] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.154s
I20250905 19:59:54.688920 30313 consensus_queue.cc:1035] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.109s
I20250905 19:59:54.644474 30100 consensus_queue.cc:1046] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.153s
I20250905 19:59:54.786738 30201 raft_consensus.cc:2802] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:54.915827 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:54.916857 28158 raft_consensus.cc:2391] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 4: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
W20250905 19:59:54.915789 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 122884us
I20250905 19:59:54.838253 28081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1397) took 2590 ms. Trace:
I20250905 19:59:54.919404 28081 rpcz_store.cc:276] 0905 19:59:52.247454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:54.807766 (+2560312us) service_pool.cc:225] Handling call
0905 19:59:54.836099 (+ 28333us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:54.837971 (+ 1872us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:54.838194 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:54.841944 28051 leader_election.cc:304] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:54.808650 28080 raft_consensus.cc:2466] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 19:59:54.836512 30213 raft_consensus.cc:695] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:54.800661 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:54.874279 28083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1393) took 2628 ms. Trace:
I20250905 19:59:54.926204 28001 raft_consensus.cc:1238] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 5. Current term is 6. Ops: []
I20250905 19:59:54.805757 30292 raft_consensus.cc:3058] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Advancing to term 7
I20250905 19:59:54.876583 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:54.926364 28083 rpcz_store.cc:276] 0905 19:59:52.246289 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:54.774365 (+2528076us) service_pool.cc:225] Handling call
0905 19:59:54.774900 (+ 535us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:54.873909 (+ 99009us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:54.874194 (+ 285us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:54.878888 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:54.840471 27894 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet fc7dc431504743d79dbdc8eea83f9dcb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 26b2f874b8e54f08b01a9ed00d5cce05 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:54.882755 30333 tablet_replica.cc:331] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05: stopping tablet replica
I20250905 19:59:54.885314 30287 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:54.888996 30093 raft_consensus.cc:513] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:54.890559 30321 log.cc:927] Time spent T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.365s user 0.000s sys 0.001s
I20250905 19:59:54.891476 28077 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50660 (request call id 225) took 1271 ms. Trace:
I20250905 19:59:54.891901 30232 leader_election.cc:290] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:54.839613 30315 log.cc:927] Time spent T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.313s user 0.000s sys 0.001s
I20250905 19:59:54.877066 30037 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:54.894201 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:54.813011 30080 consensus_queue.cc:237] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:54.896265 30318 log.cc:927] Time spent T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.308s user 0.000s sys 0.001s
I20250905 19:59:54.769733 30302 consensus_queue.cc:1035] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.111s
I20250905 19:59:54.896834 29941 raft_consensus.cc:2747] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:54.811007 30072 raft_consensus.cc:3058] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Advancing to term 6
I20250905 19:59:54.907266 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:55.017356 28005 raft_consensus.cc:3058] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:54.911844 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:54.913883 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:59:54.923786 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:55.018357 28006 raft_consensus.cc:1811] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Handling vote request from an unknown peer 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:54.797225 28079 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd"
dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
mode: GRACEFUL
new_leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:51630
I20250905 19:59:55.019341 28006 raft_consensus.cc:2366] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3.
I20250905 19:59:55.019464 28079 raft_consensus.cc:604] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Received request to transfer leadership to TS cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:55.019833 28079 raft_consensus.cc:612] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250905 19:59:54.928040 28083 raft_consensus.cc:1273] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250905 19:59:55.020417 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:55.020680 28083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1399) took 2705 ms. Trace:
I20250905 19:59:55.021037 28083 rpcz_store.cc:276] 0905 19:59:52.315465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:54.927301 (+2611836us) service_pool.cc:225] Handling call
0905 19:59:54.927640 (+ 339us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:55.020414 (+ 92774us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:55.020627 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:54.801108 29924 leader_election.cc:290] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 19:59:54.928241 27894 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 87825us
W20250905 19:59:54.925848 28717 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.26.58.131:36705 timed out after 2.337s (SENT) [suppressed 49 similar messages]
I20250905 19:59:54.874118 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:54.934494 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:54.941670 28080 raft_consensus.cc:1273] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250905 19:59:54.939596 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:55.074000 28010 raft_consensus.cc:3058] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 19:59:54.829034 30247 raft_consensus.cc:513] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:54.946828 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:54.923822 30201 raft_consensus.cc:491] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:54.931639 30100 raft_consensus.cc:3053] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:59:54.954694 29864 raft_consensus.cc:3053] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:59:54.941792 28077 rpcz_store.cc:276] 0905 19:59:53.619736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:54.535691 (+915955us) service_pool.cc:225] Handling call
0905 19:59:54.891380 (+355689us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:54.951320 28076 consensus_meta.cc:287] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.545s user 0.009s sys 0.001s
I20250905 19:59:54.963449 30333 raft_consensus.cc:2241] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:59:55.109074 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:55.109943 28154 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
I20250905 19:59:55.084846 30287 raft_consensus.cc:2802] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:54.976449 30319 leader_election.cc:290] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:55.095012 30201 raft_consensus.cc:3058] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:54.977986 30262 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:54.952837 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:55.123910 28004 raft_consensus.cc:2466] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
W20250905 19:59:55.125257 28715 leader_election.cc:336] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 3.077s (SENT)
I20250905 19:59:55.125875 28715 leader_election.cc:304] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:55.126492 28154 raft_consensus.cc:1273] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250905 19:59:54.981913 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:55.009445 28159 raft_consensus.cc:1273] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250905 19:59:55.016916 30148 raft_consensus.cc:491] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:55.032025 30289 consensus_queue.cc:260] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:54.952383 30017 consensus_meta.cc:287] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.630s user 0.008s sys 0.003s
I20250905 19:59:55.039741 28150 consensus_queue.cc:237] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:55.043268 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:55.045678 30267 leader_election.cc:290] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:55.048265 30213 consensus_queue.cc:237] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 19:59:55.049373 30279 leader_election.cc:290] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:55.050468 30165 leader_election.cc:290] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:55.167488 30232 raft_consensus.cc:491] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:55.168069 30232 raft_consensus.cc:513] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:55.170002 30232 leader_election.cc:290] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:55.170778 28010 raft_consensus.cc:2466] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 19:59:55.140890 30132 raft_consensus.cc:2747] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 19:59:55.051232 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:55.052137 30093 leader_election.cc:290] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:55.053776 30234 raft_consensus.cc:3053] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 LEADER]: Stepping down as leader of term 5
I20250905 19:59:55.055323 30317 leader_election.cc:290] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:55.056365 30299 raft_consensus.cc:3053] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 5 LEADER]: Stepping down as leader of term 5
W20250905 19:59:55.064927 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:55.065915 28005 raft_consensus.cc:2466] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
W20250905 19:59:55.066331 28151 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.549s user 0.000s sys 0.008s
W20250905 19:59:55.198244 28151 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.681s user 0.001s sys 0.008s
I20250905 19:59:55.068328 28125 leader_election.cc:304] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 19:59:55.069160 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:55.071451 27973 leader_election.cc:304] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
W20250905 19:59:55.210548 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 139204us
W20250905 19:59:55.075201 27975 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.536s user 0.031s sys 0.025s
W20250905 19:59:55.211546 27975 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.673s user 0.031s sys 0.026s
I20250905 19:59:55.073311 28080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1398) took 2773 ms. Trace:
I20250905 19:59:55.214740 28080 rpcz_store.cc:276] 0905 19:59:52.299720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:54.932034 (+2632314us) service_pool.cc:225] Handling call
0905 19:59:54.933018 (+ 984us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:55.072926 (+139908us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:55.073225 (+ 299us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:55.216619 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:55.097720 28079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:51630 (request call id 12) took 1991 ms. Trace:
I20250905 19:59:55.218806 28079 rpcz_store.cc:276] 0905 19:59:53.029183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:54.777512 (+1748329us) service_pool.cc:225] Handling call
0905 19:59:55.020160 (+242648us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:55.102519 28085 raft_consensus.cc:2466] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 19:59:55.074903 29924 raft_consensus.cc:491] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:54.973994 30322 log.cc:927] Time spent T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.351s user 0.001s sys 0.000s
W20250905 19:59:55.116391 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 274556us
I20250905 19:59:55.227674 28051 leader_election.cc:304] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:54.976984 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:55.229301 28051 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:55.230680 28051 leader_election.cc:304] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 19:59:55.095278 30100 raft_consensus.cc:738] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:55.098460 29864 raft_consensus.cc:738] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
W20250905 19:59:55.111225 30263 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.563s user 0.000s sys 0.011s
I20250905 19:59:55.142185 28006 raft_consensus.cc:2466] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
W20250905 19:59:55.142765 28715 leader_election.cc:336] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 2.544s (SENT)
W20250905 19:59:55.249393 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 106676us
I20250905 19:59:55.086913 30185 consensus_queue.cc:1035] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20250905 19:59:55.108806 28076 raft_consensus.cc:2466] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 3.
W20250905 19:59:55.155987 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 230141us
I20250905 19:59:55.159353 28159 raft_consensus.cc:1273] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 19:59:55.163650 30333 raft_consensus.cc:2270] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:59:55.164340 28077 raft_consensus.cc:1215] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Deduplicated request from leader. Original: 2.5->[4.6-4.6] Dedup: 4.6->[]
I20250905 19:59:55.296013 28077 raft_consensus.cc:1238] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250905 19:59:55.297035 28077 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 197) took 2267 ms. Trace:
I20250905 19:59:55.297390 28081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1431) took 3030 ms. Trace:
I20250905 19:59:55.297809 28081 rpcz_store.cc:276] 0905 19:59:52.267344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:54.928261 (+2660917us) service_pool.cc:225] Handling call
0905 19:59:54.946840 (+ 18579us) raft_consensus.cc:1469] Updating replica for 2 ops
0905 19:59:54.953252 (+ 6412us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:54.953283 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0905 19:59:54.953956 (+ 673us) log.cc:844] Serialized 52 byte log entry
0905 19:59:55.113682 (+159726us) raft_consensus.cc:1643] Marking committed up to 2
0905 19:59:55.113754 (+ 72us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:55.113891 (+ 137us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:55.297174 (+183283us) raft_consensus.cc:1683] finished
0905 19:59:55.297197 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:55.297332 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10735,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":131,"thread_start_us":158678,"threads_started":1,"wal-append.queue_time_us":340652}
I20250905 19:59:55.299801 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:55.181496 28003 raft_consensus.cc:3058] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
W20250905 19:59:55.195001 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 381075us
I20250905 19:59:55.153326 30017 raft_consensus.cc:513] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:55.223660 28010 raft_consensus.cc:1238] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250905 19:59:55.223976 30037 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:55.224169 29924 raft_consensus.cc:513] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:55.225047 30317 raft_consensus.cc:2747] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 19:59:55.197134 30299 raft_consensus.cc:738] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
W20250905 19:59:55.225795 30330 log.cc:927] Time spent T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
W20250905 19:59:55.226182 30166 consensus_meta.cc:287] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.649s user 0.010s sys 0.000s
W20250905 19:59:55.121415 28002 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.007s sys 0.005s
W20250905 19:59:55.345117 28002 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.732s user 0.007s sys 0.005s
W20250905 19:59:55.195660 30176 consensus_meta.cc:287] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.850s user 0.008s sys 0.011s
I20250905 19:59:55.354363 28081 raft_consensus.cc:1238] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 5 LEADER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 3. Current term is 5. Ops: []
I20250905 19:59:55.236223 28005 raft_consensus.cc:1238] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:55.356940 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:55.237659 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:55.357925 28085 raft_consensus.cc:2466] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
I20250905 19:59:55.359689 28085 raft_consensus.cc:1273] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250905 19:59:55.360420 28085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1434) took 2327 ms. Trace:
I20250905 19:59:55.360809 28085 rpcz_store.cc:276] 0905 19:59:53.032939 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:55.358837 (+2325898us) service_pool.cc:225] Handling call
0905 19:59:55.359257 (+ 420us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:55.360111 (+ 854us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:55.360344 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:55.217998 28080 raft_consensus.cc:2466] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 19:59:55.240299 27973 leader_election.cc:304] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:55.238816 28049 leader_election.cc:304] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
W20250905 19:59:55.250471 28715 leader_election.cc:336] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 2.598s (SENT)
W20250905 19:59:55.364612 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 124465us
W20250905 19:59:55.365310 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 114885us
I20250905 19:59:55.365698 27973 leader_election.cc:304] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
W20250905 19:59:55.366232 28715 leader_election.cc:336] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 3.290s (SENT)
I20250905 19:59:55.249737 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:55.251953 28150 raft_consensus.cc:1273] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250905 19:59:55.148360 30148 raft_consensus.cc:513] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:55.248445 30263 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.700s user 0.000s sys 0.011s
I20250905 19:59:55.263440 30302 raft_consensus.cc:2802] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 19:59:55.233654 30232 raft_consensus.cc:491] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:55.264572 30340 consensus_queue.cc:1035] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:55.264643 29826 consensus_queue.cc:1035] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:55.330302 30341 raft_consensus.cc:491] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:55.383351 30017 leader_election.cc:290] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:55.383922 30341 raft_consensus.cc:513] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:55.386624 30232 raft_consensus.cc:513] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:55.387156 30299 consensus_queue.cc:260] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:55.388589 30299 raft_consensus.cc:3058] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Advancing to term 6
I20250905 19:59:55.386632 30317 raft_consensus.cc:2802] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:55.389307 30317 raft_consensus.cc:491] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:55.389683 30317 raft_consensus.cc:3058] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:55.372936 30302 raft_consensus.cc:491] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:55.391407 30302 raft_consensus.cc:3058] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
W20250905 19:59:55.391688 30342 log.cc:927] Time spent T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.148s user 0.001s sys 0.000s
I20250905 19:59:55.271924 30072 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:55.348809 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:55.394837 30072 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:55.396397 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:55.194370 30234 raft_consensus.cc:738] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:55.398634 28746 raft_consensus.cc:1238] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 5. Current term is 6. Ops: []
I20250905 19:59:55.399356 30234 consensus_queue.cc:260] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:55.400602 30234 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
W20250905 19:59:55.234973 30351 log.cc:927] Time spent T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20250905 19:59:55.290503 28051 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
W20250905 19:59:55.290513 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 222248us
I20250905 19:59:55.271633 30165 raft_consensus.cc:2802] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 19:59:55.405581 30165 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:55.406415 30165 raft_consensus.cc:3058] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:55.301188 28122 leader_election.cc:304] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:55.408004 30287 raft_consensus.cc:491] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:55.222486 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:55.408807 30287 raft_consensus.cc:3058] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:55.409354 28008 raft_consensus.cc:3058] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:55.411067 30080 raft_consensus.cc:2802] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:59:55.320616 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:55.415356 28010 raft_consensus.cc:2408] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250905 19:59:55.243880 30352 raft_consensus.cc:2747] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 19:59:55.248669 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:55.332389 28751 raft_consensus.cc:1273] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250905 19:59:55.330420 28747 raft_consensus.cc:1273] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250905 19:59:55.337177 28007 raft_consensus.cc:1238] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:55.455411 30247 leader_election.cc:290] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:55.457716 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:55.348327 30100 consensus_queue.cc:260] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 19:59:55.460860 30100 raft_consensus.cc:3058] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:55.305620 30279 raft_consensus.cc:2802] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:55.372785 28001 raft_consensus.cc:3058] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
W20250905 19:59:55.473017 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 171031us
I20250905 19:59:55.373346 27973 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:55.377595 28080 raft_consensus.cc:1273] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250905 19:59:55.379385 30292 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:55.380380 30262 raft_consensus.cc:2802] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:59:55.381263 28748 raft_consensus.cc:1273] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250905 19:59:55.266589 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:55.308424 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:55.490597 28079 raft_consensus.cc:2391] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 5: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
W20250905 19:59:55.332370 30306 log.cc:927] Time spent T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.078s user 0.001s sys 0.000s
W20250905 19:59:55.307787 28076 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50660 (request call id 211) took 1947 ms (client timeout 2544 ms). Trace:
I20250905 19:59:55.419126 28081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 198) took 2389 ms. Trace:
I20250905 19:59:55.498558 28081 rpcz_store.cc:276] 0905 19:59:53.029757 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:55.324551 (+2294794us) service_pool.cc:225] Handling call
0905 19:59:55.354071 (+ 29520us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:55.418586 (+ 64515us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:55.419025 (+ 439us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:55.423200 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:55.500355 28005 raft_consensus.cc:2466] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
W20250905 19:59:55.425014 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 134658us
I20250905 19:59:55.501363 28006 raft_consensus.cc:2466] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
W20250905 19:59:55.426000 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 124895us
I20250905 19:59:55.427667 28125 leader_election.cc:304] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:55.429987 28085 raft_consensus.cc:1273] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250905 19:59:55.510740 28085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 203) took 2416 ms. Trace:
I20250905 19:59:55.511312 28085 rpcz_store.cc:276] 0905 19:59:53.094892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:55.361945 (+2267053us) service_pool.cc:225] Handling call
0905 19:59:55.429507 (+ 67562us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:55.510309 (+ 80802us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:55.510657 (+ 348us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:55.431138 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:55.374256 30166 raft_consensus.cc:513] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:55.297405 28077 rpcz_store.cc:276] 0905 19:59:53.029469 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:55.119718 (+2090249us) service_pool.cc:225] Handling call
0905 19:59:55.164179 (+ 44461us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:55.296508 (+132329us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:55.296953 (+ 445us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:55.449139 30148 leader_election.cc:290] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:55.468958 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:55.478969 28003 raft_consensus.cc:2466] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 19:59:55.467217 30072 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:55.479254 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 105997us
I20250905 19:59:55.480607 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:59:55.464519 30233 consensus_meta.cc:287] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.916s user 0.008s sys 0.005s
I20250905 19:59:55.478418 28080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 218) took 1937 ms. Trace:
I20250905 19:59:55.386322 30341 leader_election.cc:290] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:55.414516 30080 raft_consensus.cc:491] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:55.366827 29864 consensus_queue.cc:260] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:55.658845 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:55.574319 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:55.532006 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:59:55.753054 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 325471us
W20250905 19:59:55.687711 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 449069us
I20250905 19:59:55.486588 30263 raft_consensus.cc:2802] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 19:59:55.796447 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:55.796614 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:55.483646 30279 raft_consensus.cc:491] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:55.772205 27973 leader_election.cc:304] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 19:59:55.860749 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 88652.8us
I20250905 19:59:55.857959 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:59:55.772485 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:55.862267 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:55.862775 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:55.591437 28081 raft_consensus.cc:1238] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 6. Current term is 7. Ops: []
I20250905 19:59:55.773043 28085 raft_consensus.cc:1238] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 2. Current term is 3. Ops: []
W20250905 19:59:55.773461 28122 connection.cc:232] RPC call timeout handler was delayed by -1.2132970650000061s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.44630000000000003s for a response
I20250905 19:59:55.754276 28080 rpcz_store.cc:276] 0905 19:59:53.541533 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:55.362984 (+1821451us) service_pool.cc:225] Handling call
0905 19:59:55.363372 (+ 388us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:55.477982 (+114610us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:55.478336 (+ 354us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:55.481994 29941 raft_consensus.cc:2747] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:55.607000 28079 raft_consensus.cc:1273] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250905 19:59:55.602312 28084 raft_consensus.cc:2391] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 4: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:55.519856 30352 raft_consensus.cc:2802] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 19:59:55.941340 28085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 223) took 2322 ms. Trace:
I20250905 19:59:55.861776 27973 leader_election.cc:304] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
W20250905 19:59:55.955335 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 93614.1us
I20250905 19:59:55.941431 28079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 221) took 2338 ms. Trace:
I20250905 19:59:55.915307 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:56.001186 28079 rpcz_store.cc:276] 0905 19:59:53.603369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:55.516436 (+1913067us) service_pool.cc:225] Handling call
0905 19:59:55.606502 (+ 90066us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:55.941057 (+334555us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:55.941334 (+ 277us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:56.001744 27973 leader_election.cc:304] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 19:59:55.890635 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 402604us
I20250905 19:59:55.955367 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:56.004740 28154 raft_consensus.cc:3058] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
I20250905 19:59:55.615885 30232 leader_election.cc:290] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:55.994297 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:56.006603 28122 leader_election.cc:304] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:55.954291 28085 rpcz_store.cc:276] 0905 19:59:53.619081 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:55.516769 (+1897688us) service_pool.cc:225] Handling call
0905 19:59:55.772869 (+256100us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:55.941061 (+168192us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:55.941280 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:56.009052 28085 raft_consensus.cc:1273] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250905 19:59:55.866075 30279 raft_consensus.cc:3058] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:55.523241 30240 raft_consensus.cc:2747] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 19:59:56.004029 28005 raft_consensus.cc:2391] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 5: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
W20250905 19:59:55.915652 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:55.486832 30189 consensus_queue.cc:1035] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.384s
W20250905 19:59:55.907670 30280 consensus_meta.cc:287] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.228s user 0.016s sys 0.000s
W20250905 19:59:55.517513 28076 rpcz_store.cc:269] 0905 19:59:53.357989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:54.296011 (+938022us) service_pool.cc:225] Handling call
0905 19:59:55.305203 (+1009192us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:55.944320 30196 consensus_meta.cc:287] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 1.399s user 0.015s sys 0.003s
I20250905 19:59:55.792728 30080 raft_consensus.cc:3058] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:55.518110 30262 raft_consensus.cc:695] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:55.796504 30263 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:56.082922 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:55.794867 30233 raft_consensus.cc:513] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:56.016009 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:56.016296 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:59:56.019497 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:56.096024 28150 raft_consensus.cc:2408] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250905 19:59:55.798594 30072 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:55.797192 29924 leader_election.cc:290] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:55.864516 30285 consensus_queue.cc:1162] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" responder_term: 4 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: [3.2-3.2]" } } last_received_current_leader { term: 0 index: 0 } }
I20250905 19:59:56.016615 30232 raft_consensus.cc:2802] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:55.566681 30017 consensus_queue.cc:1035] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.443s
I20250905 19:59:55.687459 30176 consensus_queue.cc:1035] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.330s
I20250905 19:59:56.017338 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:56.016629 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:56.028126 28154 raft_consensus.cc:2466] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
I20250905 19:59:56.114992 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:56.116349 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:56.028537 28085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1435) took 1554 ms. Trace:
W20250905 19:59:56.028893 28009 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s user 0.014s sys 0.035s
I20250905 19:59:56.009052 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:56.117168 28154 raft_consensus.cc:2391] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 5: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
W20250905 19:59:56.117571 28009 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.653s user 0.014s sys 0.035s
I20250905 19:59:56.117319 28085 rpcz_store.cc:276] 0905 19:59:54.456084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:56.008380 (+1552296us) service_pool.cc:225] Handling call
0905 19:59:56.008704 (+ 324us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:56.010172 (+ 1468us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:56.010385 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:56.117936 28003 raft_consensus.cc:2391] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 6: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:55.921186 30333 ts_tablet_manager.cc:1905] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 19:59:55.994701 30352 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:56.133764 30352 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Advancing to term 8
I20250905 19:59:55.565292 29995 consensus_queue.cc:1035] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.455s
I20250905 19:59:55.911545 29864 raft_consensus.cc:3058] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:55.860030 30341 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:56.151129 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:56.082206 28082 raft_consensus.cc:2466] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 19:59:56.088285 28005 raft_consensus.cc:3058] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
W20250905 19:59:56.075981 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:56.103513 29924 raft_consensus.cc:491] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:56.103946 30292 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:56.017385 30185 raft_consensus.cc:2747] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 19:59:55.863996 28081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 222) took 2245 ms. Trace:
I20250905 19:59:56.157845 28081 rpcz_store.cc:276] 0905 19:59:53.618525 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:55.520496 (+1901971us) service_pool.cc:225] Handling call
0905 19:59:55.591180 (+ 70684us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:55.863648 (+272468us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:55.863926 (+ 278us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:56.110010 27975 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a
W20250905 19:59:56.110535 28155 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.811s user 0.036s sys 0.044s
I20250905 19:59:56.015156 30319 raft_consensus.cc:2802] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 19:59:56.059964 30148 raft_consensus.cc:2802] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:56.128278 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:56.167447 28154 raft_consensus.cc:2391] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 5: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:56.168529 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:56.169432 28154 raft_consensus.cc:2391] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 4: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:56.130710 28157 raft_consensus.cc:1238] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:56.094734 30263 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Advancing to term 8
W20250905 19:59:56.038697 30350 log.cc:927] Time spent T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.147s user 0.000s sys 0.001s
I20250905 19:59:56.144990 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:56.105299 30232 raft_consensus.cc:491] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:56.037495 30280 raft_consensus.cc:513] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:56.113279 30166 leader_election.cc:290] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:56.030253 30366 consensus_queue.cc:1035] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.130s
W20250905 19:59:56.160053 28155 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.860s user 0.037s sys 0.044s
I20250905 19:59:56.153975 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:56.165655 28008 consensus_meta.cc:287] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.754s user 0.010s sys 0.002s
I20250905 19:59:56.145555 30267 raft_consensus.cc:491] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:56.178330 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:56.178537 28150 raft_consensus.cc:2391] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 6: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
W20250905 19:59:56.179086 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:56.183442 28005 raft_consensus.cc:2466] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 19:59:56.206812 28008 raft_consensus.cc:2466] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
I20250905 19:59:56.182729 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:56.179615 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:56.209399 28081 raft_consensus.cc:3058] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 19:59:56.210146 28005 raft_consensus.cc:1238] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 5. Current term is 6. Ops: []
W20250905 19:59:56.201781 28001 consensus_meta.cc:287] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.728s user 0.008s sys 0.001s
I20250905 19:59:56.162411 30292 raft_consensus.cc:491] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:56.211771 28001 raft_consensus.cc:2466] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 19:59:56.213413 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:56.214323 28085 raft_consensus.cc:2391] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 4: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:56.215253 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:56.216361 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:56.218577 28746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40926 (request call id 444) took 1090 ms. Trace:
I20250905 19:59:56.218139 30267 raft_consensus.cc:513] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:56.161904 30319 raft_consensus.cc:491] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:56.166914 30148 raft_consensus.cc:491] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:56.220922 28153 raft_consensus.cc:2391] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 4: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:56.156111 29924 raft_consensus.cc:513] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:56.246116 28158 raft_consensus.cc:2391] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 4: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:56.187134 30232 raft_consensus.cc:3058] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:56.156363 30341 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:56.219002 28746 rpcz_store.cc:276] 0905 19:59:55.128635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:55.131132 (+ 2497us) service_pool.cc:225] Handling call
0905 19:59:55.163723 (+ 32591us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:55.399232 (+235509us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:56.177861 (+778629us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b540062b1d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:56.218483 (+ 40622us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":539519360}
I20250905 19:59:56.220280 28076 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1442) took 1423 ms. Trace:
I20250905 19:59:56.255402 30372 raft_consensus.cc:2802] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 19:59:56.240248 30319 raft_consensus.cc:3058] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:56.257138 30267 leader_election.cc:290] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:56.229370 30292 raft_consensus.cc:513] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:56.288863 30292 leader_election.cc:290] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:56.260391 28002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 272) took 1669 ms. Trace:
I20250905 19:59:56.289568 28002 rpcz_store.cc:276] 0905 19:59:54.591093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:54.591262 (+ 169us) service_pool.cc:225] Handling call
0905 19:59:54.597879 (+ 6617us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:54.611901 (+ 14022us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:54.611941 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:54.612492 (+ 551us) log.cc:844] Serialized 26 byte log entry
0905 19:59:55.345739 (+733247us) raft_consensus.cc:1643] Marking committed up to 5
0905 19:59:55.345805 (+ 66us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:55.345917 (+ 112us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:56.260147 (+914230us) raft_consensus.cc:1683] finished
0905 19:59:56.260174 (+ 27us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:56.260326 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2962,"raft.queue_time_us":14905,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":19914,"thread_start_us":732781,"threads_started":1,"wal-append.queue_time_us":991233}
I20250905 19:59:56.292610 28002 raft_consensus.cc:1238] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 5. Current term is 6. Ops: []
I20250905 19:59:56.255453 30280 leader_election.cc:290] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:56.272588 28076 rpcz_store.cc:276] 0905 19:59:54.780051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:56.088893 (+1308842us) service_pool.cc:225] Handling call
0905 19:59:56.143562 (+ 54669us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:56.155380 (+ 11818us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:56.157278 (+ 1898us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:56.163607 (+ 6329us) raft_consensus.cc:1643] Marking committed up to 3
0905 19:59:56.168362 (+ 4755us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:56.184399 (+ 16037us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:56.203498 (+ 19099us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:56.275626 30213 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:56.299175 28076 raft_consensus.cc:1273] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250905 19:59:56.299340 30213 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:56.299865 28076 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1450) took 1058 ms. Trace:
I20250905 19:59:56.300222 28076 rpcz_store.cc:276] 0905 19:59:55.241286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:56.297770 (+1056484us) service_pool.cc:225] Handling call
0905 19:59:56.298407 (+ 637us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:56.299561 (+ 1154us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:56.299805 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:56.301512 30213 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:56.275461 30372 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:56.302263 30372 raft_consensus.cc:3058] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
W20250905 19:59:56.263772 30363 log.cc:927] Time spent T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
W20250905 19:59:56.269953 30201 consensus_meta.cc:287] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 1.129s user 0.013s sys 0.000s
W20250905 19:59:56.271103 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 264578us
I20250905 19:59:56.256985 29924 leader_election.cc:290] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:56.243144 30148 raft_consensus.cc:3058] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:56.206240 30364 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:59:56.285948 30302 consensus_meta.cc:287] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.894s user 0.010s sys 0.000s
W20250905 19:59:56.275650 30349 log.cc:927] Time spent T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.478s user 0.001s sys 0.001s
W20250905 19:59:56.354142 30287 consensus_meta.cc:287] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.945s user 0.013s sys 0.000s
I20250905 19:59:56.331773 28081 raft_consensus.cc:2466] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6.
I20250905 19:59:56.343670 28158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39586 (request call id 261) took 1559 ms. Trace:
I20250905 19:59:56.194571 30262 consensus_queue.cc:237] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:56.348510 30359 log.cc:927] Time spent T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.660s user 0.000s sys 0.003s
I20250905 19:59:56.352890 30232 raft_consensus.cc:513] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 19:59:56.353693 30369 log.cc:927] Time spent T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.083s user 0.000s sys 0.006s
I20250905 19:59:56.316805 28076 raft_consensus.cc:3058] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:56.354851 28083 raft_consensus.cc:1273] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20250905 19:59:56.355235 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:56.355140 28006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1971) took 1123 ms. Trace:
I20250905 19:59:56.390113 28006 rpcz_store.cc:276] 0905 19:59:55.232114 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:55.249380 (+ 17266us) service_pool.cc:225] Handling call
0905 19:59:56.354879 (+1105499us) spinlock_profiling.cc:243] Waited 7.3 ms on lock 0x7b540034c454. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:56.355048 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16792960}
W20250905 19:59:56.356637 28001 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1374) took 1411 ms (client timeout 1641 ms). Trace:
W20250905 19:59:56.391883 28001 rpcz_store.cc:269] 0905 19:59:54.945190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:54.981539 (+ 36349us) service_pool.cc:225] Handling call
0905 19:59:56.356423 (+1374884us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b54002c0454. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:56.356561 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":562736000}
I20250905 19:59:56.360667 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:56.367280 28158 rpcz_store.cc:276] 0905 19:59:54.784734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:54.934046 (+149312us) service_pool.cc:225] Handling call
0905 19:59:56.343276 (+1409230us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54000ebfd4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:56.343533 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2691904384}
I20250905 19:59:56.320760 30201 raft_consensus.cc:513] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:56.345741 30319 raft_consensus.cc:513] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:56.320010 28122 leader_election.cc:304] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 19:59:56.362421 30100 consensus_meta.cc:287] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.901s user 0.009s sys 0.001s
W20250905 19:59:56.374562 28010 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.768s user 0.025s sys 0.006s
W20250905 19:59:56.365144 30234 consensus_meta.cc:287] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.964s user 0.013s sys 0.002s
I20250905 19:59:56.384891 28080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 236) took 2618 ms. Trace:
I20250905 19:59:56.368489 30289 raft_consensus.cc:3058] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:56.425025 28157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2040) took 1975 ms. Trace:
I20250905 19:59:56.425307 28078 raft_consensus.cc:1238] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:56.398159 28079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 242) took 2629 ms. Trace:
W20250905 19:59:56.394441 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:56.425513 28157 rpcz_store.cc:276] 0905 19:59:54.449202 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:54.517713 (+ 68511us) service_pool.cc:225] Handling call
0905 19:59:54.518054 (+ 341us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:56.169822 (+1651768us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:56.424722 (+254900us) spinlock_profiling.cc:243] Waited 1.61 s on lock 0x7b5400316d54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:56.424953 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3708106112}
I20250905 19:59:56.417099 28080 rpcz_store.cc:276] 0905 19:59:53.766588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:55.942549 (+2175961us) service_pool.cc:225] Handling call
0905 19:59:55.942955 (+ 406us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:56.376925 (+433970us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:56.376973 (+ 48us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:56.384455 (+ 7482us) raft_consensus.cc:1643] Marking committed up to 4
0905 19:59:56.384524 (+ 69us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:56.384622 (+ 98us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:56.384810 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":70654,"raft.queue_time_us":241334,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":41573,"thread_start_us":366242,"threads_started":1}
I20250905 19:59:56.368745 30233 leader_election.cc:290] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 19:59:56.426617 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:56.415282 28010 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.809s user 0.026s sys 0.006s
I20250905 19:59:56.364495 30364 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:56.420292 28007 raft_consensus.cc:1238] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20250905 19:59:56.420739 28159 raft_consensus.cc:2466] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
I20250905 19:59:56.372104 30287 raft_consensus.cc:513] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:56.426126 28079 rpcz_store.cc:276] 0905 19:59:53.768898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:56.002864 (+2233966us) service_pool.cc:225] Handling call
0905 19:59:56.029599 (+ 26735us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:56.397583 (+367984us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:56.397631 (+ 48us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:56.397697 (+ 66us) raft_consensus.cc:1643] Marking committed up to 4
0905 19:59:56.397756 (+ 59us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:56.397861 (+ 105us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:56.398068 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5651,"raft.queue_time_us":165285,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":24904,"thread_start_us":367423,"threads_started":1}
W20250905 19:59:56.410650 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 90776.5us
W20250905 19:59:56.425530 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 388494us
I20250905 19:59:56.413873 30374 raft_consensus.cc:491] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:56.365780 30302 raft_consensus.cc:513] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:56.456791 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 346845us
W20250905 19:59:56.418529 30299 consensus_meta.cc:287] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 1.029s user 0.005s sys 0.010s
W20250905 19:59:56.451943 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:56.450134 28122 leader_election.cc:304] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:56.416265 30213 raft_consensus.cc:2802] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250905 19:59:56.442581 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 432828us
W20250905 19:59:56.453751 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 293743us
I20250905 19:59:56.458185 27975 leader_election.cc:304] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:56.465901 28122 leader_election.cc:304] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:56.473263 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:56.473897 28157 raft_consensus.cc:3058] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Advancing to term 7
I20250905 19:59:56.474664 28122 leader_election.cc:400] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Vote denied by peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193) with higher term. Message: Invalid argument: T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3.
I20250905 19:59:56.474812 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:56.475023 28122 leader_election.cc:403] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250905 19:59:56.475576 28158 raft_consensus.cc:2391] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 4: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
W20250905 19:59:56.475538 28122 raft_consensus.cc:2684] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250905 19:59:56.476640 28122 leader_election.cc:304] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:56.476583 30267 consensus_queue.cc:1035] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:56.480552 28085 raft_consensus.cc:1273] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20250905 19:59:56.482026 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:56.482686 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:56.480607 27975 leader_election.cc:304] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:56.476665 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:56.488903 28158 raft_consensus.cc:2391] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 4: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:56.489682 27975 leader_election.cc:304] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:56.490063 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
W20250905 19:59:56.454677 30317 consensus_meta.cc:287] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 1.065s user 0.009s sys 0.004s
I20250905 19:59:56.483937 28152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39586 (request call id 262) took 1692 ms. Trace:
I20250905 19:59:56.500648 28152 rpcz_store.cc:276] 0905 19:59:54.791260 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:55.191374 (+400114us) service_pool.cc:225] Handling call
0905 19:59:55.192069 (+ 695us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:55.204028 (+ 11959us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:55.204077 (+ 49us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:55.245446 (+ 41369us) log.cc:844] Serialized 26 byte log entry
0905 19:59:55.496066 (+250620us) spinlock_profiling.cc:243] Waited 2.88 ms on lock 0x7b180041c3e0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49ac86d47 00007fe4a38dcb99 00007fe49aca369b 00007fe49aca193f 00007fe49aca3c02 00007fe49d8572bb 00007fe49d85cdc3 00007fe49d85d171 00007fe49dbec09f
0905 19:59:55.895314 (+399248us) raft_consensus.cc:1643] Marking committed up to 4
0905 19:59:55.895375 (+ 61us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:55.895476 (+ 101us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:56.460021 (+564545us) raft_consensus.cc:1683] finished
0905 19:59:56.460042 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:56.483865 (+ 23823us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36836,"raft.queue_time_us":11930,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":23808,"spinlock_wait_cycles":6624512,"thread_start_us":398688,"threads_started":1,"wal-append.queue_time_us":874533}
I20250905 19:59:56.484422 27973 leader_election.cc:304] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:56.455708 28744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40904 (request call id 557) took 1164 ms. Trace:
I20250905 19:59:56.483736 28747 raft_consensus.cc:2466] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 19:59:56.492336 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:56.504343 28080 raft_consensus.cc:3058] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:56.493515 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:56.505897 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:56.506680 28081 raft_consensus.cc:2391] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 4: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
W20250905 19:59:56.481956 30165 consensus_meta.cc:287] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 1.075s user 0.013s sys 0.000s
I20250905 19:59:56.495800 28751 raft_consensus.cc:2391] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 4: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 19:59:56.480280 30234 raft_consensus.cc:2747] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 19:59:56.480104 30280 raft_consensus.cc:2747] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 19:59:56.481307 30213 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:56.492841 27975 leader_election.cc:304] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:56.483726 30292 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:56.459255 30233 consensus_queue.cc:1046] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.057s
I20250905 19:59:56.516984 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:56.522073 28009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1377) took 1488 ms. Trace:
I20250905 19:59:56.483616 30340 consensus_queue.cc:1046] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.033s
I20250905 19:59:56.522868 27975 leader_election.cc:304] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:56.525306 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:56.525978 28009 rpcz_store.cc:276] 0905 19:59:55.034153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:55.055293 (+ 21140us) service_pool.cc:225] Handling call
0905 19:59:55.055685 (+ 392us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:55.463949 (+408264us) raft_consensus.cc:1532] Early marking committed up to index 2
0905 19:59:55.464425 (+ 476us) log.cc:844] Serialized 14 byte log entry
0905 19:59:56.418021 (+953596us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:56.420662 (+ 2641us) raft_consensus.cc:1643] Marking committed up to 2
0905 19:59:56.420714 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:56.521727 (+101013us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b5400290c54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:56.521795 (+ 68us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:56.521994 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23890,"prepare.queue_time_us":8692,"prepare.run_cpu_time_us":117,"prepare.run_wall_time_us":18212,"raft.queue_time_us":130141,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":2672,"spinlock_wait_cycles":937857920,"thread_start_us":953125,"threads_started":2,"wal-append.queue_time_us":694289}
I20250905 19:59:56.518782 30341 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:56.479663 30240 consensus_queue.cc:1035] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20250905 19:59:56.514865 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:56.460347 30374 raft_consensus.cc:513] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:56.518080 30213 raft_consensus.cc:3058] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:56.538236 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:56.541301 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:56.539006 28079 raft_consensus.cc:2466] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
W20250905 19:59:56.531929 30362 log.cc:927] Time spent T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.202s user 0.001s sys 0.001s
I20250905 19:59:56.511643 30317 raft_consensus.cc:513] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:56.529543 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 19:59:56.545859 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 366525us
W20250905 19:59:56.547394 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 424850us
I20250905 19:59:56.511294 28744 rpcz_store.cc:276] 0905 19:59:55.288144 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:55.301050 (+ 12906us) service_pool.cc:225] Handling call
0905 19:59:56.360466 (+1059416us) spinlock_profiling.cc:243] Waited 9.52 ms on lock 0x7b5800304598. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b7db60 00007fe4a2b7d444 00007fe49c2050c4 00007fe49c205052
0905 19:59:56.452251 (+ 91785us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21896064}
W20250905 19:59:56.594938 28082 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1428) took 2356 ms (client timeout 2199 ms). Trace:
I20250905 19:59:56.540414 30341 raft_consensus.cc:2747] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250905 19:59:56.595358 28082 rpcz_store.cc:269] 0905 19:59:54.238116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:54.700753 (+462637us) service_pool.cc:225] Handling call
0905 19:59:56.594715 (+1893962us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b54001266d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:56.594858 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2974692224}
W20250905 19:59:56.596841 30279 consensus_meta.cc:287] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.557s user 0.011s sys 0.000s
I20250905 19:59:56.597249 30279 raft_consensus.cc:513] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:56.598347 28078 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1430) took 4331 ms. Trace:
I20250905 19:59:56.598802 28156 raft_consensus.cc:1238] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:56.598773 28078 rpcz_store.cc:276] 0905 19:59:52.266976 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:54.831831 (+2564855us) service_pool.cc:225] Handling call
0905 19:59:54.882978 (+ 51147us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:56.425926 (+1542948us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:56.598014 (+172088us) spinlock_profiling.cc:243] Waited 1.54 s on lock 0x7b54003aa554. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:56.598284 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3546427648}
I20250905 19:59:56.599470 30279 leader_election.cc:290] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:56.600076 30319 leader_election.cc:290] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:56.600605 28083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1406) took 3628 ms. Trace:
I20250905 19:59:56.601008 28083 rpcz_store.cc:276] 0905 19:59:52.971957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:55.077369 (+2105412us) service_pool.cc:225] Handling call
0905 19:59:55.077807 (+ 438us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:56.389228 (+1311421us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:56.600364 (+211136us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b5400127ad4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:56.600545 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2935919872}
I20250905 19:59:56.510324 28157 raft_consensus.cc:2466] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 7.
I20250905 19:59:56.536727 30267 consensus_queue.cc:1035] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20250905 19:59:56.579013 30232 leader_election.cc:290] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:56.581007 28010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1969) took 1367 ms. Trace:
I20250905 19:59:56.514833 30165 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:56.528904 30292 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:56.584321 30263 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:56.585538 30287 leader_election.cc:290] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:56.588930 30317 leader_election.cc:290] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:56.589450 28002 raft_consensus.cc:1238] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 3. Current term is 4. Ops: []
W20250905 19:59:56.591701 28084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1425) took 2354 ms (client timeout 1696 ms). Trace:
I20250905 19:59:56.594058 30100 raft_consensus.cc:3053] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:59:56.607758 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:56.614241 28010 rpcz_store.cc:276] 0905 19:59:55.214045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:55.474470 (+260425us) service_pool.cc:225] Handling call
0905 19:59:55.474878 (+ 408us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:55.475246 (+ 368us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:55.475273 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:55.475716 (+ 443us) log.cc:844] Serialized 26 byte log entry
0905 19:59:56.427929 (+952213us) raft_consensus.cc:1643] Marking committed up to 2
0905 19:59:56.462686 (+ 34757us) log.cc:844] Serialized 14 byte log entry
0905 19:59:56.463083 (+ 397us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:56.463187 (+ 104us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:56.580776 (+117589us) raft_consensus.cc:1683] finished
0905 19:59:56.580800 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:56.580931 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1671,"prepare.queue_time_us":52098,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":1473,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":10141,"spinlock_wait_cycles":8031616,"thread_start_us":821419,"threads_started":1,"wal-append.queue_time_us":1033576}
I20250905 19:59:56.615869 28156 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2047) took 1549 ms. Trace:
I20250905 19:59:56.640458 28156 rpcz_store.cc:276] 0905 19:59:55.066152 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:55.300987 (+234835us) service_pool.cc:225] Handling call
0905 19:59:55.328593 (+ 27606us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:56.599470 (+1270877us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:56.615333 (+ 15863us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54000ec9d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:56.615732 (+ 399us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2920582400}
I20250905 19:59:56.621094 28747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40926 (request call id 448) took 1153 ms. Trace:
I20250905 19:59:56.642393 28747 rpcz_store.cc:276] 0905 19:59:55.467796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:55.468586 (+ 790us) service_pool.cc:225] Handling call
0905 19:59:56.620854 (+1152268us) spinlock_profiling.cc:243] Waited 693 ms on lock 0x7b54005415d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:56.621006 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1593345792}
I20250905 19:59:56.623435 28085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1447) took 1417 ms. Trace:
I20250905 19:59:56.644261 28085 rpcz_store.cc:276] 0905 19:59:55.205743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:56.217155 (+1011412us) service_pool.cc:225] Handling call
0905 19:59:56.217451 (+ 296us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:56.481142 (+263691us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:56.613694 (+132552us) spinlock_profiling.cc:243] Waited 263 ms on lock 0x7b540019e6d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:56.623350 (+ 9656us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":604316032}
W20250905 19:59:56.625484 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:56.637283 28084 rpcz_store.cc:269] 0905 19:59:54.237198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:54.351096 (+113898us) service_pool.cc:225] Handling call
0905 19:59:56.591452 (+2240356us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b540021fa54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:56.591619 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2874513664}
I20250905 19:59:56.607831 28050 leader_election.cc:304] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:56.608593 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:56.648725 28079 raft_consensus.cc:2408] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20250905 19:59:56.609380 28049 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.821s user 0.002s sys 0.020s
W20250905 19:59:56.653295 28049 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.865s user 0.002s sys 0.020s
I20250905 19:59:56.610772 28004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1379) took 1575 ms. Trace:
I20250905 19:59:56.660373 28081 raft_consensus.cc:2408] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250905 19:59:56.662520 28051 leader_election.cc:304] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:56.621297 30341 raft_consensus.cc:2802] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election won for term 6
I20250905 19:59:56.620043 30201 leader_election.cc:290] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:56.613514 29924 raft_consensus.cc:2747] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250905 19:59:56.663132 30080 consensus_meta.cc:287] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.555s user 0.011s sys 0.001s
I20250905 19:59:56.656883 30232 raft_consensus.cc:2802] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 19:59:56.660962 30379 raft_consensus.cc:2802] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader election won for term 7
I20250905 19:59:56.664359 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:56.662732 30317 raft_consensus.cc:2747] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 19:59:56.639187 30100 raft_consensus.cc:738] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:56.614113 30383 consensus_queue.cc:1035] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.095s
W20250905 19:59:56.703516 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 45541.5us
I20250905 19:59:56.699786 28155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2049) took 1493 ms. Trace:
I20250905 19:59:56.701432 30364 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:56.737391 28155 rpcz_store.cc:276] 0905 19:59:55.206794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:55.294608 (+ 87814us) service_pool.cc:225] Handling call
0905 19:59:55.295052 (+ 444us) raft_consensus.cc:1469] Updating replica for 2 ops
0905 19:59:55.298922 (+ 3870us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:55.298955 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0905 19:59:55.299525 (+ 570us) log.cc:844] Serialized 52 byte log entry
0905 19:59:55.412631 (+113106us) spinlock_profiling.cc:243] Waited 2.92 ms on lock 0x7b48000cd158. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a2ae2c51 00007fe4a2ad9df7 00007fe4a2ad9c8a 00007fe4a2bcf07f 00007fe4a2bced47 00007fe4a2be336c 00007fe4a2be3318 00007fe4a2be3298 00007fe4a2be3253 00007fe4a2be24fc 00007fe4a1b606f6
0905 19:59:56.203856 (+791225us) raft_consensus.cc:1643] Marking committed up to 7
0905 19:59:56.676640 (+472784us) log.cc:844] Serialized 14 byte log entry
0905 19:59:56.681268 (+ 4628us) log.cc:844] Serialized 14 byte log entry
0905 19:59:56.683851 (+ 2583us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:56.683941 (+ 90us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:56.699396 (+ 15455us) raft_consensus.cc:1683] finished
0905 19:59:56.699509 (+ 113us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:56.699722 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7847,"prepare.queue_time_us":440926,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":80,"raft.queue_time_us":9430,"raft.run_cpu_time_us":2914,"raft.run_wall_time_us":138498,"spinlock_wait_cycles":55452160,"thread_start_us":1336035,"threads_started":2,"wal-append.queue_time_us":876097}
I20250905 19:59:56.686529 30185 raft_consensus.cc:2747] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 19:59:56.736660 30385 consensus_queue.cc:1035] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.010s
I20250905 19:59:56.747416 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 19:59:56.723968 28125 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.933s user 0.000s sys 0.016s
I20250905 19:59:56.709609 30232 raft_consensus.cc:491] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:56.748287 28080 raft_consensus.cc:2466] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 19:59:56.704409 30341 raft_consensus.cc:695] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:56.686815 30386 raft_consensus.cc:2747] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250905 19:59:56.753818 28125 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.963s user 0.002s sys 0.016s
I20250905 19:59:56.701715 30302 leader_election.cc:290] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:56.763794 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 102488us
I20250905 19:59:56.743528 30379 raft_consensus.cc:695] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 7 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
W20250905 19:59:56.767347 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 980506us
W20250905 19:59:56.762382 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 107237us
I20250905 19:59:56.723121 30080 raft_consensus.cc:513] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:56.772293 30364 raft_consensus.cc:2747] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:56.659058 28004 rpcz_store.cc:276] 0905 19:59:55.034977 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:55.124773 (+ 89796us) service_pool.cc:225] Handling call
0905 19:59:55.160785 (+ 36012us) raft_consensus.cc:1469] Updating replica for 2 ops
0905 19:59:55.183780 (+ 22995us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:55.183821 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0905 19:59:55.228840 (+ 45019us) log.cc:844] Serialized 52 byte log entry
0905 19:59:55.521428 (+292588us) raft_consensus.cc:1643] Marking committed up to 2
0905 19:59:56.171317 (+649889us) log.cc:844] Serialized 14 byte log entry
0905 19:59:56.610281 (+438964us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:56.610394 (+ 113us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:56.610483 (+ 89us) raft_consensus.cc:1683] finished
0905 19:59:56.610506 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:56.610694 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":81248,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":13688,"raft.queue_time_us":86849,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":203,"spinlock_wait_cycles":3328,"thread_start_us":1068648,"threads_started":3,"wal-append.queue_time_us":884936}
I20250905 19:59:56.772380 30232 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Advancing to term 7
I20250905 19:59:56.748878 30201 consensus_queue.cc:1046] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
W20250905 19:59:56.811131 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 326737us
I20250905 19:59:56.811764 28151 raft_consensus.cc:1238] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:56.777779 30148 raft_consensus.cc:513] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:56.787204 30374 leader_election.cc:290] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:56.821835 28125 leader_election.cc:400] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Vote denied by peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421) with higher term. Message: Invalid argument: T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 2. Current term is 3.
W20250905 19:59:56.809387 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 322235us
I20250905 19:59:56.755519 30289 consensus_queue.cc:1035] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
W20250905 19:59:56.806062 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 315153us
I20250905 19:59:56.779078 28051 leader_election.cc:304] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:56.830629 28125 leader_election.cc:304] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 19:59:56.833361 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 355053us
I20250905 19:59:56.836913 27973 leader_election.cc:304] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:56.766857 30100 consensus_queue.cc:260] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:56.857060 28125 leader_election.cc:304] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:56.858284 28125 leader_election.cc:304] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:56.855234 28718 leader_election.cc:304] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:56.860332 30391 consensus_queue.cc:1035] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 19:59:56.861325 27973 leader_election.cc:304] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:56.862154 30389 raft_consensus.cc:2802] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 19:59:56.862632 30389 raft_consensus.cc:491] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:56.820436 30302 raft_consensus.cc:3053] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Stepping down as leader of term 3
W20250905 19:59:56.864805 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 342030us
W20250905 19:59:56.867750 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 88768.5us
I20250905 19:59:56.823259 30263 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:56.871901 27975 leader_election.cc:304] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:56.867013 28125 leader_election.cc:304] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:56.868489 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:56.873183 27975 leader_election.cc:304] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:56.870189 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:56.866837 27973 leader_election.cc:304] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:56.872583 28051 leader_election.cc:304] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 19:59:56.882959 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 45695.1us
W20250905 19:59:56.889340 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:56.883985 28122 leader_election.cc:304] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:56.858891 30383 raft_consensus.cc:2762] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20250905 19:59:56.858335 30292 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:56.877785 28005 raft_consensus.cc:1238] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 LEADER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 2. Current term is 3. Ops: []
W20250905 19:59:56.905434 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:56.881109 28051 leader_election.cc:304] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 19:59:56.943804 28717 leader_election.cc:336] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Timed out: RequestConsensusVote RPC to 127.26.58.131:36705 timed out after 2.516s (SENT)
W20250905 19:59:56.944444 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 63452.4us
I20250905 19:59:56.939949 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:56.946285 28051 leader_election.cc:304] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 19:59:56.948561 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 390407us
I20250905 19:59:56.949852 28051 leader_election.cc:304] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:56.866984 30302 raft_consensus.cc:738] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:56.951647 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:56.955488 28080 raft_consensus.cc:2374] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5. Re-sending same reply.
W20250905 19:59:56.953106 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 86184.6us
W20250905 19:59:56.953554 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 69690.1us
I20250905 19:59:56.863579 30374 consensus_queue.cc:1035] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.069s
I20250905 19:59:56.866811 30389 raft_consensus.cc:3058] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 19:59:56.875929 30106 raft_consensus.cc:2747] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 19:59:56.876044 28076 raft_consensus.cc:1273] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20250905 19:59:56.958611 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 85537.2us
I20250905 19:59:56.995247 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:56.933461 30100 raft_consensus.cc:3058] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:56.994491 28085 raft_consensus.cc:1238] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250905 19:59:56.962720 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:56.881709 29964 raft_consensus.cc:2747] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 19:59:57.004251 28080 raft_consensus.cc:3058] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 19:59:56.995543 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:57.005280 28082 raft_consensus.cc:2466] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:56.968185 28125 leader_election.cc:304] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a
W20250905 19:59:56.972018 28122 connection.cc:232] RPC call timeout handler was delayed by -1.2118522399999847s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1696s for a response
W20250905 19:59:57.007332 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:57.003707 27975 leader_election.cc:304] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:56.884727 30196 raft_consensus.cc:2747] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 19:59:56.875416 30386 consensus_queue.cc:1035] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.117s
I20250905 19:59:56.933904 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 19:59:56.950476 30352 consensus_meta.cc:287] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.793s user 0.010s sys 0.003s
I20250905 19:59:56.881908 30319 raft_consensus.cc:2802] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election won for term 6
I20250905 19:59:56.949764 28077 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 232) took 3328 ms. Trace:
I20250905 19:59:57.050954 28080 raft_consensus.cc:2466] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
W20250905 19:59:56.963008 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 96239.3us
I20250905 19:59:56.875562 30388 raft_consensus.cc:2747] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:56.960297 30397 raft_consensus.cc:491] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:56.872325 30391 consensus_queue.cc:1046] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.101s
I20250905 19:59:56.952637 30201 raft_consensus.cc:2747] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 19:59:56.874976 30385 consensus_queue.cc:1035] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.109s
I20250905 19:59:57.014025 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:56.952817 30292 raft_consensus.cc:2747] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 19:59:57.023428 30384 raft_consensus.cc:491] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
W20250905 19:59:57.031160 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 176005us
I20250905 19:59:56.994860 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:56.866184 30387 consensus_queue.cc:1035] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.124s
I20250905 19:59:56.962502 30393 raft_consensus.cc:2802] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election won for term 5
I20250905 19:59:56.963192 30280 raft_consensus.cc:2802] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election won for term 6
I20250905 19:59:56.963932 30335 raft_consensus.cc:2802] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:59:56.961553 30364 raft_consensus.cc:2747] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 19:59:56.993610 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:56.828073 30341 consensus_queue.cc:237] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:57.047264 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:57.014158 30165 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:57.143115 28076 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 245) took 1011 ms. Trace:
I20250905 19:59:57.251839 28151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1500) took 1297 ms. Trace:
I20250905 19:59:57.320750 28005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 282) took 1299 ms. Trace:
I20250905 19:59:57.050205 30352 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:56.900211 30233 consensus_queue.cc:1035] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.110s
I20250905 19:59:56.993268 30262 raft_consensus.cc:3058] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:56.926940 30379 consensus_queue.cc:237] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:57.367384 28078 raft_consensus.cc:2466] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 19:59:56.977145 30289 raft_consensus.cc:2802] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:57.014875 30396 raft_consensus.cc:491] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
W20250905 19:59:57.517733 28152 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.011s sys 0.033s
W20250905 19:59:57.527128 28152 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.510s user 0.011s sys 0.033s
I20250905 19:59:57.394919 30400 raft_consensus.cc:2747] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 19:59:56.996153 30234 raft_consensus.cc:3053] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 LEADER]: Stepping down as leader of term 5
I20250905 19:59:57.375705 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:56.870496 30390 consensus_queue.cc:1035] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.135s
I20250905 19:59:56.995811 30263 raft_consensus.cc:2802] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:57.376406 28082 raft_consensus.cc:1238] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20250905 19:59:57.377310 28002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 284) took 1341 ms. Trace:
I20250905 19:59:57.046736 29924 raft_consensus.cc:491] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
W20250905 19:59:57.379317 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 375656us
I20250905 19:59:57.428495 30080 leader_election.cc:290] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:57.382397 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 19:59:57.383530 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 415411us
I20250905 19:59:57.385195 28715 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:57.389706 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:57.379379 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:57.403862 28083 raft_consensus.cc:3058] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:57.453971 28151 rpcz_store.cc:276] 0905 19:59:55.954848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:55.955622 (+ 774us) service_pool.cc:225] Handling call
0905 19:59:55.995323 (+ 39701us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:56.817338 (+822015us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:57.251492 (+434154us) spinlock_profiling.cc:243] Waited 816 ms on lock 0x7b5400326c54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:57.251751 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1876960512}
I20250905 19:59:57.456461 28005 rpcz_store.cc:276] 0905 19:59:56.021002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:56.232109 (+211107us) service_pool.cc:225] Handling call
0905 19:59:56.232520 (+ 411us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:56.938339 (+705819us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:57.320408 (+382069us) spinlock_profiling.cc:243] Waited 645 ms on lock 0x7b54003831d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:57.320656 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1483310976}
I20250905 19:59:57.387411 30176 raft_consensus.cc:491] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
W20250905 19:59:57.392095 30232 consensus_meta.cc:287] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.576s user 0.012s sys 0.000s
I20250905 19:59:57.507138 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:57.509775 28085 raft_consensus.cc:2391] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 4: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:57.578635 30176 raft_consensus.cc:513] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:57.579912 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:57.580811 30176 leader_election.cc:290] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:57.581261 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:57.582214 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 164531us
W20250905 19:59:57.511029 30404 log.cc:927] Time spent T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.142s user 0.002s sys 0.000s
I20250905 19:59:57.512267 30385 consensus_queue.cc:1046] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250905 19:59:57.520174 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:57.522042 28157 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:36164
W20250905 19:59:57.524793 28122 leader_election.cc:336] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.696s (SENT)
W20250905 19:59:57.525300 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:56.994141 30302 consensus_queue.cc:260] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:57.594707 28125 leader_election.cc:304] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 19:59:57.595875 28125 leader_election.cc:304] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 19:59:57.596093 30412 log.cc:927] Time spent T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.189s user 0.000s sys 0.001s
I20250905 19:59:57.394783 30346 consensus_queue.cc:1046] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.173s
W20250905 19:59:57.530560 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 167685us
I20250905 19:59:57.567806 30289 raft_consensus.cc:695] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
W20250905 19:59:57.598693 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:57.600172 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:57.599553 30289 consensus_queue.cc:237] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:57.601040 30267 consensus_queue.cc:1046] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:59:57.602125 30420 raft_consensus.cc:2802] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:57.581804 29924 raft_consensus.cc:513] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:57.602097 30267 raft_consensus.cc:2802] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:57.604526 30267 raft_consensus.cc:695] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:57.605322 29924 leader_election.cc:290] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:57.605662 30267 consensus_queue.cc:237] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:57.562355 30398 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.026s sys 0.090s
I20250905 19:59:57.536957 30280 raft_consensus.cc:695] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:57.565886 30234 raft_consensus.cc:738] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
W20250905 19:59:57.509764 30395 log.cc:927] Time spent T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.534s user 0.007s sys 0.003s
I20250905 19:59:57.562649 28002 rpcz_store.cc:276] 0905 19:59:56.035605 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:56.293672 (+258067us) service_pool.cc:225] Handling call
0905 19:59:56.294044 (+ 372us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:56.636929 (+342885us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:57.252580 (+615651us) spinlock_profiling.cc:243] Waited 295 ms on lock 0x7b54001315d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:57.377243 (+124663us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":678872320}
W20250905 19:59:57.634069 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:57.568470 27975 leader_election.cc:304] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:57.055065 28745 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 8.
W20250905 19:59:57.431245 30394 log.cc:927] Time spent T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.483s user 0.002s sys 0.009s
I20250905 19:59:57.651304 28003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1980) took 2148 ms. Trace:
W20250905 19:59:57.652170 30408 log.cc:927] Time spent T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20250905 19:59:57.602982 30420 raft_consensus.cc:695] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
W20250905 19:59:57.540421 30213 consensus_meta.cc:287] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.990s user 0.008s sys 0.007s
I20250905 19:59:57.654520 28008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 277) took 1634 ms. Trace:
I20250905 19:59:57.412871 30384 raft_consensus.cc:513] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:57.654768 30213 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:57.655099 30420 consensus_queue.cc:237] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:57.654986 28008 rpcz_store.cc:276] 0905 19:59:56.019622 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:56.208140 (+188518us) service_pool.cc:225] Handling call
0905 19:59:56.208716 (+ 576us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:56.638682 (+429966us) raft_consensus.cc:1532] Early marking committed up to index 6
0905 19:59:56.659550 (+ 20868us) log.cc:844] Serialized 14 byte log entry
0905 19:59:56.865038 (+205488us) log.cc:844] Serialized 14 byte log entry
0905 19:59:56.956571 (+ 91533us) spinlock_profiling.cc:243] Waited 3.75 ms on lock 0x7b1800090ac0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49ac19f4a 00007fe4a4107409 00007fe49aadb3e0 00007fe49aca14e4 00007fe49aca1455 00007fe49dc11020 00007fe49dc236c6 00007fe49dc34237 00007fe49dc341b8
0905 19:59:56.956773 (+ 202us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:56.991504 (+ 34731us) log.cc:844] Serialized 26 byte log entry
0905 19:59:56.991775 (+ 271us) raft_consensus.cc:1643] Marking committed up to 6
0905 19:59:56.991842 (+ 67us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:56.991954 (+ 112us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:57.654143 (+662189us) raft_consensus.cc:1683] finished
0905 19:59:57.654178 (+ 35us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:57.654392 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":66292,"prepare.queue_time_us":426,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":84,"raft.queue_time_us":145263,"raft.run_cpu_time_us":337,"raft.run_wall_time_us":39295,"spinlock_wait_cycles":8628096,"thread_start_us":581813,"threads_started":2,"wal-append.queue_time_us":847600}
I20250905 19:59:57.657155 30384 leader_election.cc:290] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:57.657382 30213 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 19:59:57.583029 28051 connection.cc:232] RPC call timeout handler was delayed by -1.0411635409999462s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2846s for a response
W20250905 19:59:57.585314 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 200186us
I20250905 19:59:57.621073 30072 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:57.659749 30072 consensus_queue.cc:1035] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250905 19:59:57.661159 28049 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
W20250905 19:59:57.536917 29864 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.522s user 0.012s sys 0.021s
I20250905 19:59:57.662271 28049 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:57.663208 29941 raft_consensus.cc:2747] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250905 19:59:57.663410 30196 raft_consensus.cc:2802] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250905 19:59:57.577415 28081 raft_consensus.cc:3058] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:57.664561 28150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2055) took 1192 ms. Trace:
I20250905 19:59:57.530283 30148 leader_election.cc:290] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:57.591353 28079 raft_consensus.cc:1238] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20250905 19:59:57.651847 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:57.592381 28157 raft_consensus.cc:604] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 LEADER]: Received request to transfer leadership to TS cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:57.593461 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 19:59:57.651819 28003 rpcz_store.cc:276] 0905 19:59:55.503036 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:56.119610 (+616574us) service_pool.cc:225] Handling call
0905 19:59:56.207297 (+ 87687us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:56.670710 (+463413us) raft_consensus.cc:1532] Early marking committed up to index 3
0905 19:59:56.863791 (+193081us) log.cc:844] Serialized 14 byte log entry
0905 19:59:57.380411 (+516620us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:57.380491 (+ 80us) raft_consensus.cc:1643] Marking committed up to 3
0905 19:59:57.380563 (+ 72us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:57.380638 (+ 75us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:57.651128 (+270490us) spinlock_profiling.cc:243] Waited 56 ms on lock 0x7b5400055fd0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:57.651242 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":258176,"prepare.queue_time_us":62480,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":426851,"raft.run_cpu_time_us":401,"raft.run_wall_time_us":23442,"spinlock_wait_cycles":128723712,"thread_start_us":798942,"threads_started":3,"wal-append.queue_time_us":589120}
I20250905 19:59:57.531123 30393 raft_consensus.cc:695] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
W20250905 19:59:57.610055 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 85305.4us
I20250905 19:59:57.610144 28083 raft_consensus.cc:2466] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
W20250905 19:59:57.671454 28122 leader_election.cc:336] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 2.199s (SENT)
I20250905 19:59:57.534634 30383 raft_consensus.cc:3053] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 19:59:57.397200 30375 raft_consensus.cc:491] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:57.602948 30185 raft_consensus.cc:3053] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:59:57.634632 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 19:59:57.635385 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 67011.8us
I20250905 19:59:57.691744 28084 raft_consensus.cc:3058] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:57.692034 27975 leader_election.cc:304] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:57.528142 30333 ts_tablet_manager.cc:1918] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250905 19:59:57.454149 28076 rpcz_store.cc:276] 0905 19:59:56.117972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:56.301073 (+183101us) service_pool.cc:225] Handling call
0905 19:59:56.301410 (+ 337us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:57.010368 (+708958us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:57.129575 (+119207us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:57.534929 30396 raft_consensus.cc:513] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:57.638826 29924 raft_consensus.cc:2747] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 19:59:57.639642 28153 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2054) took 1167 ms. Trace:
I20250905 19:59:57.625661 30263 raft_consensus.cc:695] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:57.640533 30299 raft_consensus.cc:3053] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 6 LEADER]: Stepping down as leader of term 6
I20250905 19:59:57.693517 30419 raft_consensus.cc:491] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:57.727416 30080 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:57.727804 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:57.727988 30080 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:57.728560 28155 raft_consensus.cc:2466] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6.
I20250905 19:59:57.727716 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:57.729749 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 19:59:57.641306 30372 consensus_meta.cc:287] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 1.338s user 0.018s sys 0.003s
I20250905 19:59:57.644582 30280 consensus_queue.cc:237] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:57.647876 30389 consensus_meta.cc:287] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.653s user 0.013s sys 0.003s
I20250905 19:59:57.648362 30406 raft_consensus.cc:491] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:57.649396 30335 raft_consensus.cc:491] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:57.650352 30352 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:57.394608 30397 raft_consensus.cc:513] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:57.051329 28077 rpcz_store.cc:276] 0905 19:59:53.621779 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:55.566417 (+1944638us) service_pool.cc:225] Handling call
0905 19:59:55.790870 (+224453us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:55.908438 (+117568us) raft_consensus.cc:1532] Early marking committed up to index 5
0905 19:59:56.448073 (+539635us) log.cc:844] Serialized 14 byte log entry
0905 19:59:56.867046 (+418973us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:56.870211 (+ 3165us) raft_consensus.cc:1643] Marking committed up to 5
0905 19:59:56.883229 (+ 13018us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:56.901073 (+ 17844us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:56.941770 (+ 40697us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27191,"prepare.queue_time_us":782312,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":1749,"raft.queue_time_us":38191,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":6441,"thread_start_us":670782,"threads_started":2,"wal-append.queue_time_us":480820}
I20250905 19:59:57.657766 30384 consensus_queue.cc:1035] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250905 19:59:57.388762 30106 raft_consensus.cc:2802] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250905 19:59:57.616755 30398 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.619s user 0.027s sys 0.090s
I20250905 19:59:57.663975 30196 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:59:57.664966 28150 rpcz_store.cc:276] 0905 19:59:56.472259 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:56.477254 (+ 4995us) service_pool.cc:225] Handling call
0905 19:59:56.477545 (+ 291us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:56.477943 (+ 398us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:56.477971 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:56.478502 (+ 531us) log.cc:844] Serialized 26 byte log entry
0905 19:59:56.749630 (+271128us) raft_consensus.cc:1643] Marking committed up to 3
0905 19:59:56.749672 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:56.749742 (+ 70us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:57.509345 (+759603us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b2400181358. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49aaf9784 00007fe49aaf788b 00007fe49ab86669 00007fe49ab862f5 00007fe49ab8593d 00007fe49d84120e 00007fe49d85893c 00007fe49d857c66 00007fe49d857888 00007fe49d8612c1
0905 19:59:57.664276 (+154931us) raft_consensus.cc:1683] finished
0905 19:59:57.664299 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:57.664496 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":702,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":140,"spinlock_wait_cycles":23367808,"thread_start_us":267984,"threads_started":1,"wal-append.queue_time_us":386233}
W20250905 19:59:57.696282 28051 leader_election.cc:336] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.657s (SENT)
I20250905 19:59:57.701696 28083 raft_consensus.cc:1238] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20250905 19:59:57.727221 28156 raft_consensus.cc:1238] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 6. Current term is 7. Ops: [4.6-6.7]
I20250905 19:59:57.731097 28152 raft_consensus.cc:1238] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 5. Current term is 6. Ops: []
I20250905 19:59:57.734906 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:57.733641 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:57.704530 27975 leader_election.cc:304] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:57.730016 29995 raft_consensus.cc:491] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:57.746294 28081 raft_consensus.cc:2466] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 19:59:57.752009 30335 raft_consensus.cc:3058] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
W20250905 19:59:57.762988 30409 log.cc:927] Time spent T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.243s user 0.000s sys 0.004s
I20250905 19:59:57.749174 30389 raft_consensus.cc:513] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:57.731822 28158 raft_consensus.cc:1238] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 5. Current term is 6. Ops: []
I20250905 19:59:57.747851 28080 raft_consensus.cc:2466] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 19:59:57.792169 28159 raft_consensus.cc:2391] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 4: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:57.793249 28080 raft_consensus.cc:1238] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: []
W20250905 19:59:57.793928 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 99507us
W20250905 19:59:57.748674 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 70976.9us
I20250905 19:59:57.729712 28153 rpcz_store.cc:276] 0905 19:59:56.471966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:56.476749 (+ 4783us) service_pool.cc:225] Handling call
0905 19:59:56.477075 (+ 326us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:56.552347 (+ 75272us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:56.552385 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:56.553007 (+ 622us) log.cc:844] Serialized 26 byte log entry
0905 19:59:56.830705 (+277698us) raft_consensus.cc:1643] Marking committed up to 2
0905 19:59:56.830758 (+ 53us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:57.244105 (+413347us) spinlock_profiling.cc:243] Waited 74.8 ms on lock 0x7b54000ed3d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:57.244171 (+ 66us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:57.510689 (+266518us) spinlock_profiling.cc:243] Waited 6.49 ms on lock 0x7b2400181358. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49aaf9784 00007fe49aaf788b 00007fe49ab86669 00007fe49ab862f5 00007fe49ab8593d 00007fe49d84120e 00007fe49d85893c 00007fe49d857c66 00007fe49d857888 00007fe49d8612c1
0905 19:59:57.639422 (+128733us) raft_consensus.cc:1683] finished
0905 19:59:57.639449 (+ 27us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:57.639585 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1127,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":23887,"spinlock_wait_cycles":186882688,"thread_start_us":249651,"threads_started":1,"wal-append.queue_time_us":430196}
I20250905 19:59:57.745266 30346 raft_consensus.cc:2747] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250905 19:59:57.684996 30383 raft_consensus.cc:738] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:57.776743 30080 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:57.711967 30416 raft_consensus.cc:3053] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 LEADER]: Stepping down as leader of term 5
I20250905 19:59:57.835350 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:57.691316 30185 raft_consensus.cc:738] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:57.841673 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:57.609462 30232 raft_consensus.cc:513] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:57.766878 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 374605us
W20250905 19:59:57.849907 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:57.710808 30148 raft_consensus.cc:3058] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:57.731806 30166 raft_consensus.cc:491] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:57.723207 29941 raft_consensus.cc:491] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:57.744751 30299 raft_consensus.cc:738] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:57.868933 28076 raft_consensus.cc:1273] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250905 19:59:57.726842 29964 raft_consensus.cc:491] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:57.783913 28051 leader_election.cc:304] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:57.746740 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 19:59:57.893834 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 490639us
I20250905 19:59:57.823733 28122 leader_election.cc:304] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:57.895347 28157 raft_consensus.cc:2391] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 4: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
W20250905 19:59:57.896068 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 72422.5us
I20250905 19:59:57.824494 30402 raft_consensus.cc:491] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:57.824882 28084 raft_consensus.cc:2466] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 19:59:57.723802 30393 consensus_queue.cc:237] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:57.751293 30240 consensus_queue.cc:1035] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20250905 19:59:57.779963 30106 raft_consensus.cc:491] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:57.727698 30419 raft_consensus.cc:513] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:57.787351 30196 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Advancing to term 9
I20250905 19:59:57.834230 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:57.836263 28007 raft_consensus.cc:2391] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:57.837801 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:57.732210 30234 consensus_queue.cc:260] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:57.851733 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:57.853987 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:57.849519 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:57.702481 30375 raft_consensus.cc:513] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:57.867221 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:57.836970 30213 consensus_queue.cc:1046] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.073s
I20250905 19:59:57.821916 29924 raft_consensus.cc:2747] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:57.822633 30317 raft_consensus.cc:491] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:57.714654 30333 log.cc:1199] T 986c4f475a8d46f39465775db70416d3 P 26b2f874b8e54f08b01a9ed00d5cce05: Deleting WAL directory at /tmp/dist-test-taskX7vhwv/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1757102265724956-26858-0/minicluster-data/ts-2-root/wals/986c4f475a8d46f39465775db70416d3
I20250905 19:59:57.750535 30372 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:57.902384 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:57.825865 30371 raft_consensus.cc:491] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:57.849002 30319 raft_consensus.cc:695] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:57.835486 30416 raft_consensus.cc:738] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:57.793968 29995 raft_consensus.cc:513] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
W20250905 19:59:57.911223 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 214966us
I20250905 19:59:57.893779 30166 raft_consensus.cc:513] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:57.849061 30017 consensus_queue.cc:1046] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.062s
I20250905 19:59:57.929392 28009 raft_consensus.cc:2391] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 4: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 19:59:57.929703 28010 raft_consensus.cc:2366] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4.
I20250905 19:59:57.901858 29964 raft_consensus.cc:513] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:57.889178 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:57.897285 30080 raft_consensus.cc:2747] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:59:57.934653 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:57.906512 30302 raft_consensus.cc:3058] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:57.965716 30234 raft_consensus.cc:3058] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
I20250905 19:59:57.905622 30398 raft_consensus.cc:491] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:57.959020 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:57.922019 29864 raft_consensus.cc:2747] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:59:57.923007 30106 raft_consensus.cc:3058] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
W20250905 19:59:57.967376 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 352167us
W20250905 19:59:57.970312 28051 leader_election.cc:336] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.734s (SENT)
I20250905 19:59:57.759720 30406 raft_consensus.cc:513] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:57.927400 28077 raft_consensus.cc:1273] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250905 19:59:57.980213 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:57.930411 28006 raft_consensus.cc:2466] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 19:59:57.894045 29941 raft_consensus.cc:513] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:57.868639 30263 consensus_queue.cc:237] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:57.992067 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 19:59:57.995301 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:57.995523 28159 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
from {username='slave'} at 127.0.0.1:36164
I20250905 19:59:57.950129 28001 raft_consensus.cc:1238] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: []
I20250905 19:59:58.010563 28083 raft_consensus.cc:1273] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20250905 19:59:57.960899 30317 raft_consensus.cc:513] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:57.967264 30415 raft_consensus.cc:491] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:58.013306 28008 raft_consensus.cc:2391] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 4: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:57.953009 30384 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 19:59:57.926944 30185 consensus_queue.cc:260] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:57.933326 30299 consensus_queue.cc:260] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:57.912267 30402 raft_consensus.cc:513] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:57.966040 30371 raft_consensus.cc:513] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:58.020686 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 113213us
I20250905 19:59:58.460039 28051 leader_election.cc:304] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:58.024533 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:58.460227 28010 raft_consensus.cc:2391] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:58.460258 28159 raft_consensus.cc:604] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Received request to transfer leadership to TS e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:58.461895 28009 raft_consensus.cc:2366] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 2. Current term is 4.
W20250905 19:59:58.113075 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:58.460327 28083 pending_rounds.cc:85] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9: Aborting all ops after (but not including) 2
W20250905 19:59:58.030669 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:58.463376 28083 pending_rounds.cc:107] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20250905 19:59:58.463855 28083 raft_consensus.cc:2887] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250905 19:59:58.215426 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:58.465484 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:58.465699 30292 raft_consensus.cc:491] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:58.462134 28159 raft_consensus.cc:612] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250905 19:59:58.469352 28717 leader_election.cc:304] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:58.466264 30292 raft_consensus.cc:513] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:58.489537 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:58.493096 28009 raft_consensus.cc:2366] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 6. Current term is 7.
W20250905 19:59:58.424472 30423 log.cc:927] Time spent T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.120s user 0.000s sys 0.000s
W20250905 19:59:58.489243 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 518914us
W20250905 19:59:58.493638 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:58.502355 28051 leader_election.cc:336] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.778s (SENT)
W20250905 19:59:58.503312 28051 leader_election.cc:336] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.793s (SENT)
W20250905 19:59:58.154039 30440 log.cc:927] Time spent T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.133s user 0.000s sys 0.000s
W20250905 19:59:58.415421 30433 log.cc:927] Time spent T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410: Append to log took a long time: real 0.449s user 0.000s sys 0.002s
W20250905 19:59:58.099567 30428 log.cc:927] Time spent T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.165s user 0.000s sys 0.000s
W20250905 19:59:58.163985 30395 log.cc:927] Time spent T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20250905 19:59:58.467418 30017 raft_consensus.cc:491] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:58.094993 30289 consensus_queue.cc:1046] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.404s
I20250905 19:59:58.498607 30100 raft_consensus.cc:2747] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:58.039053 30398 raft_consensus.cc:513] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:58.472016 30267 raft_consensus.cc:491] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:58.500500 28153 raft_consensus.cc:1273] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250905 19:59:58.417797 30431 raft_consensus.cc:491] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:58.163021 30420 raft_consensus.cc:3053] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Stepping down as leader of term 4
I20250905 19:59:58.159008 30389 leader_election.cc:290] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:58.110972 29964 leader_election.cc:290] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:58.417538 30372 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:58.431816 30375 leader_election.cc:290] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:58.096580 30333 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33742 (request call id 674) took 3427 ms. Trace:
I20250905 19:59:58.153003 30419 leader_election.cc:290] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:58.460126 29995 leader_election.cc:290] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421)
I20250905 19:59:58.441174 30166 leader_election.cc:290] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:58.109284 30396 leader_election.cc:290] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:58.518257 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:58.664448 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:58.494318 30415 raft_consensus.cc:513] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:58.666719 30415 leader_election.cc:290] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:58.109661 30416 consensus_queue.cc:260] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:58.417738 30319 consensus_queue.cc:237] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:58.549784 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:58.548677 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:58.668015 28745 raft_consensus.cc:2466] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 19:59:58.472028 30384 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:58.666841 30292 leader_election.cc:290] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:58.158342 30397 leader_election.cc:290] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:58.671535 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:58.496083 30185 raft_consensus.cc:3058] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
W20250905 19:59:58.607553 30196 consensus_meta.cc:287] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.653s user 0.010s sys 0.000s
W20250905 19:59:58.634090 30262 consensus_meta.cc:287] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 1.107s user 0.010s sys 0.002s
W20250905 19:59:58.588150 30436 log.cc:927] Time spent T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.602s user 0.014s sys 0.004s
W20250905 19:59:58.612950 30335 consensus_meta.cc:287] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.791s user 0.009s sys 0.003s
I20250905 19:59:58.601635 30441 raft_consensus.cc:491] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:58.516593 30213 raft_consensus.cc:491] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:58.591218 30240 raft_consensus.cc:2953] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 4 to 6, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 19:59:58.670676 30420 raft_consensus.cc:738] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 19:59:58.162889 30446 consensus_queue.cc:1046] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.559s
I20250905 19:59:58.590534 30406 leader_election.cc:290] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:58.542886 30232 leader_election.cc:290] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:58.549882 30402 leader_election.cc:290] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:58.666663 30267 raft_consensus.cc:513] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:58.256681 27973 leader_election.cc:304] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 19:59:58.472851 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:58.490177 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:58.498366 28080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1463) took 1039 ms. Trace:
I20250905 19:59:58.503830 28051 leader_election.cc:304] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 19:59:58.001446 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 19:59:58.771474 28006 raft_consensus.cc:2366] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4.
I20250905 19:59:58.770205 28004 raft_consensus.cc:2391] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 5: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 19:59:58.724159 30381 raft_consensus.cc:3053] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:59:58.845579 30381 raft_consensus.cc:738] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:58.845697 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:58.848531 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:58.847868 30381 consensus_queue.cc:260] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:58.675818 30371 leader_election.cc:290] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:58.685583 30431 raft_consensus.cc:513] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:58.851495 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:58.802337 28076 raft_consensus.cc:2391] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 5: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:58.766934 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:58.719159 30317 leader_election.cc:290] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:58.680160 30451 raft_consensus.cc:2802] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 19:59:58.861879 30451 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:58.774576 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:58.862900 30451 raft_consensus.cc:3058] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Advancing to term 7
I20250905 19:59:58.768419 28003 raft_consensus.cc:1273] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20250905 19:59:58.764169 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 507451us
I20250905 19:59:58.871198 28003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 1986) took 1039 ms. Trace:
I20250905 19:59:58.871663 28003 rpcz_store.cc:276] 0905 19:59:57.832190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:57.832386 (+ 196us) service_pool.cc:225] Handling call
0905 19:59:57.890347 (+ 57961us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:58.869971 (+979624us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:58.870811 (+ 840us) spinlock_profiling.cc:243] Waited 877 ms on lock 0x7b540034c454. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:58.871121 (+ 310us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2016123264}
I20250905 19:59:58.872206 27973 leader_election.cc:304] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 19:59:58.677903 30398 leader_election.cc:290] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:58.708976 30397 raft_consensus.cc:3053] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:59:58.876492 30397 raft_consensus.cc:738] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 19:59:58.877925 28004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1994) took 1026 ms. Trace:
W20250905 19:59:58.679172 30439 log.cc:927] Time spent T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.639s user 0.005s sys 0.013s
W20250905 19:59:58.712308 30302 consensus_meta.cc:287] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.699s user 0.011s sys 0.000s
I20250905 19:59:58.682840 28747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40904 (request call id 565) took 1733 ms. Trace:
W20250905 19:59:58.674193 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 66115.4us
I20250905 19:59:58.673797 28749 raft_consensus.cc:1238] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 5. Current term is 6. Ops: []
I20250905 19:59:58.703073 30441 raft_consensus.cc:513] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:58.686086 30017 raft_consensus.cc:513] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:58.683758 30262 raft_consensus.cc:2747] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250905 19:59:58.692888 30234 consensus_meta.cc:287] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.679s user 0.008s sys 0.002s
I20250905 19:59:58.703305 30213 raft_consensus.cc:513] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 19:59:58.801122 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 297699us
I20250905 19:59:58.879715 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:58.886530 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:58.772578 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:58.800967 28083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1450) took 1600 ms. Trace:
I20250905 19:59:58.880218 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:58.878404 28004 rpcz_store.cc:276] 0905 19:59:57.851708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:57.987480 (+135772us) service_pool.cc:225] Handling call
0905 19:59:58.877623 (+890143us) spinlock_profiling.cc:243] Waited 307 ms on lock 0x7b54003840d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:58.877831 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":707125120}
I20250905 19:59:58.795972 28080 rpcz_store.cc:276] 0905 19:59:57.458799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:57.794509 (+335710us) service_pool.cc:225] Handling call
0905 19:59:57.806348 (+ 11839us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:57.806684 (+ 336us) raft_consensus.cc:1532] Early marking committed up to index 4
0905 19:59:57.831347 (+ 24663us) log.cc:844] Serialized 14 byte log entry
0905 19:59:58.492942 (+661595us) log.cc:844] Serialized 14 byte log entry
0905 19:59:58.497985 (+ 5043us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:58.498043 (+ 58us) raft_consensus.cc:1643] Marking committed up to 4
0905 19:59:58.498098 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:58.498171 (+ 73us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:58.498301 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":485590,"prepare.queue_time_us":504226,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":99,"raft.queue_time_us":23400,"raft.run_cpu_time_us":319,"raft.run_wall_time_us":17789,"thread_start_us":188717,"threads_started":1,"wal-append.queue_time_us":612350}
I20250905 19:59:58.881614 28747 rpcz_store.cc:276] 0905 19:59:56.949929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:56.950097 (+ 168us) service_pool.cc:225] Handling call
0905 19:59:56.950465 (+ 368us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:56.998577 (+ 48112us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:56.998609 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:57.013000 (+ 14391us) log.cc:844] Serialized 26 byte log entry
0905 19:59:57.745063 (+732063us) raft_consensus.cc:1643] Marking committed up to 3
0905 19:59:57.745104 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:57.745288 (+ 184us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:59:58.682520 (+937232us) raft_consensus.cc:1683] finished
0905 19:59:58.682554 (+ 34us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:58.682759 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35617,"raft.queue_time_us":180420,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":42169,"thread_start_us":289409,"threads_started":1,"wal-append.queue_time_us":799572}
I20250905 19:59:58.684454 30299 raft_consensus.cc:3058] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Advancing to term 7
I20250905 19:59:58.891909 30100 raft_consensus.cc:491] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:58.861907 29924 raft_consensus.cc:2747] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250905 19:59:58.915160 28006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1996) took 1028 ms. Trace:
I20250905 19:59:58.914893 30100 raft_consensus.cc:513] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:58.915577 28006 rpcz_store.cc:276] 0905 19:59:57.886558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:58.472561 (+586003us) service_pool.cc:225] Handling call
0905 19:59:58.903989 (+431428us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:58.916852 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:58.917625 28006 raft_consensus.cc:2374] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5. Re-sending same reply.
I20250905 19:59:58.917737 30100 leader_election.cc:290] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:58.677666 30196 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:58.672792 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:58.906337 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:58.771250 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:58.918187 28006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1999) took 1015 ms. Trace:
I20250905 19:59:58.922643 28006 rpcz_store.cc:276] 0905 19:59:57.902659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:58.916450 (+1013791us) service_pool.cc:225] Handling call
0905 19:59:58.918106 (+ 1656us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:58.924557 28008 raft_consensus.cc:2466] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
I20250905 19:59:58.924916 30196 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 9 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:58.925328 28008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 2001) took 1011 ms. Trace:
I20250905 19:59:58.925755 28008 rpcz_store.cc:276] 0905 19:59:57.914220 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:58.473803 (+559583us) service_pool.cc:225] Handling call
0905 19:59:58.925247 (+451444us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:58.926120 28079 raft_consensus.cc:2391] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 9: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 19:59:58.905390 28083 rpcz_store.cc:276] 0905 19:59:57.200147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:57.784464 (+584317us) service_pool.cc:225] Handling call
0905 19:59:57.815348 (+ 30884us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:58.464423 (+649075us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:58.800159 (+335736us) spinlock_profiling.cc:243] Waited 194 ms on lock 0x7b54003a9154. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:58.800746 (+ 587us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":446084992}
I20250905 19:59:58.927479 28079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50660 (request call id 253) took 1235 ms. Trace:
I20250905 19:59:58.927039 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:58.927924 28079 rpcz_store.cc:276] 0905 19:59:57.692496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:57.729475 (+ 36979us) service_pool.cc:225] Handling call
0905 19:59:58.927189 (+1197714us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b540019e6d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:58.927398 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2712397184}
I20250905 19:59:58.928586 28008 raft_consensus.cc:2391] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 4: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:58.920553 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:58.929877 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:58.797652 28077 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1449) took 1754 ms. Trace:
I20250905 19:59:58.879158 30381 raft_consensus.cc:3058] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Advancing to term 4
W20250905 19:59:58.714174 30148 consensus_meta.cc:287] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.822s user 0.014s sys 0.000s
I20250905 19:59:58.957319 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:58.878365 30397 consensus_queue.cc:260] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:58.963891 30434 raft_consensus.cc:491] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:58.883237 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:58.954376 28003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 1998) took 1033 ms. Trace:
I20250905 19:59:58.972623 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:58.962931 30432 raft_consensus.cc:491] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:58.976112 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:58.882894 30431 leader_election.cc:290] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:58.683933 29964 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:58.697408 30106 raft_consensus.cc:513] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:58.978821 30232 raft_consensus.cc:491] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 19:59:58.979067 30148 raft_consensus.cc:2747] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250905 19:59:58.679999 30437 consensus_queue.cc:1162] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" responder_term: 6 status { last_received { term: 5 index: 7 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 5. Current term is 6. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250905 19:59:58.981686 28003 rpcz_store.cc:276] 0905 19:59:57.900648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:58.873523 (+972875us) service_pool.cc:225] Handling call
0905 19:59:58.934177 (+ 60654us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:58.978154 30262 raft_consensus.cc:2802] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election won for term 5
I20250905 19:59:58.983609 28153 raft_consensus.cc:3058] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
I20250905 19:59:58.984277 30397 raft_consensus.cc:3058] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:58.981328 28001 raft_consensus.cc:2466] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
W20250905 19:59:58.976728 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 104667us
I20250905 19:59:58.984903 30434 raft_consensus.cc:513] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:58.990911 28006 raft_consensus.cc:3058] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:59.008047 30434 leader_election.cc:290] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:59.008284 29941 leader_election.cc:290] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:59.009114 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:58.708853 30384 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:59.010618 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:58.670858 30333 rpcz_store.cc:276] 0905 19:59:54.669016 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:54.669316 (+ 300us) service_pool.cc:225] Handling call
0905 19:59:54.820403 (+151087us) ts_tablet_manager.cc:1145] Deleting tablet 986c4f475a8d46f39465775db70416d3
0905 19:59:54.832448 (+ 12045us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:59:56.612756 (+1780308us) tablet_metadata.cc:641] Metadata flushed
0905 19:59:57.506054 (+893298us) tablet_metadata.cc:641] Metadata flushed
0905 19:59:58.096489 (+590435us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4821,"tablet-delete.queue_time_us":115618,"thread_start_us":27644,"threads_started":1}
I20250905 19:59:58.996004 28008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 2005) took 1001 ms. Trace:
I20250905 19:59:58.727187 30267 leader_election.cc:290] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:58.998944 28010 raft_consensus.cc:3058] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:59:58.988009 28076 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1469) took 1162 ms. Trace:
I20250905 19:59:59.004240 28001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1395) took 1139 ms. Trace:
W20250905 19:59:59.005617 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:59.009337 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:58.890924 30375 raft_consensus.cc:2747] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 19:59:59.012864 28008 rpcz_store.cc:276] 0905 19:59:57.994982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:58.929491 (+934509us) service_pool.cc:225] Handling call
0905 19:59:58.995914 (+ 66423us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:58.990041 30432 raft_consensus.cc:513] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:59.003821 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:59.033052 28001 rpcz_store.cc:276] 0905 19:59:57.865457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:58.879269 (+1013812us) service_pool.cc:225] Handling call
0905 19:59:59.004181 (+124912us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:59.048971 30432 leader_election.cc:290] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:59.051115 30371 raft_consensus.cc:491] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:59.052202 30371 raft_consensus.cc:513] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:59.052680 27894 catalog_manager.cc:4928] TS 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): tablet 986c4f475a8d46f39465775db70416d3 (table test-workload [id=e9f8c661c3d64feebbf330b6de7735b1]) successfully deleted
I20250905 19:59:59.054586 30406 raft_consensus.cc:491] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:59.055855 30406 raft_consensus.cc:513] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:59.058300 28156 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Advancing to term 7
I20250905 19:59:59.058930 30389 raft_consensus.cc:491] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 19:59:59.060338 28153 raft_consensus.cc:2466] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
I20250905 19:59:59.059899 30389 raft_consensus.cc:513] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:59.068331 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:59.074517 30402 raft_consensus.cc:491] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:59.077661 30441 leader_election.cc:290] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:59.004220 28051 leader_election.cc:336] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.799s (SENT)
I20250905 19:59:59.078989 29964 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:59.110368 30416 raft_consensus.cc:3058] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Advancing to term 6
I20250905 19:59:59.129324 29964 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:59.080003 30213 leader_election.cc:290] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 19:59:59.080422 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:59.131446 28078 raft_consensus.cc:2466] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 19:59:59.131889 28003 raft_consensus.cc:2391] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:59.132653 28003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 2000) took 1223 ms. Trace:
W20250905 19:59:59.132944 28078 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1454) took 1761 ms (client timeout 2080 ms). Trace:
I20250905 19:59:59.120946 30389 leader_election.cc:290] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 19:59:59.080708 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:59.133438 28078 rpcz_store.cc:269] 0905 19:59:57.371489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:57.506843 (+135354us) service_pool.cc:225] Handling call
0905 19:59:59.132746 (+1625903us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b540019ebd4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:59.132885 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2993890560}
I20250905 19:59:59.136075 28078 raft_consensus.cc:1238] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 8. Current term is 9. Ops: [6.7-8.8]
I20250905 19:59:59.136812 28078 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1468) took 1313 ms. Trace:
I20250905 19:59:59.137190 28078 rpcz_store.cc:276] 0905 19:59:57.823543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:59.135262 (+1311719us) service_pool.cc:225] Handling call
0905 19:59:59.135749 (+ 487us) raft_consensus.cc:1469] Updating replica for 2 ops
0905 19:59:59.136480 (+ 731us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:59.136704 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:59.137017 29941 consensus_queue.cc:1035] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20250905 19:59:59.120275 28749 raft_consensus.cc:1273] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250905 19:59:59.125509 30402 raft_consensus.cc:513] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:59.147759 28004 raft_consensus.cc:2408] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250905 19:59:59.149349 28004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1391) took 1299 ms. Trace:
I20250905 19:59:59.126346 28051 leader_election.cc:304] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 19:59:59.153168 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 198996us
I20250905 19:59:59.154948 30292 raft_consensus.cc:2747] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:59:59.156775 30402 leader_election.cc:290] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:59.091187 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:59.158286 28001 raft_consensus.cc:2391] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 8: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:59.159726 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:59.160773 28001 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 7.
I20250905 19:59:59.090728 30232 raft_consensus.cc:513] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:59.092056 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:58.707458 30420 consensus_queue.cc:260] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:58.702646 30335 raft_consensus.cc:513] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:58.968470 28077 rpcz_store.cc:276] 0905 19:59:57.042974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:57.781895 (+738921us) service_pool.cc:225] Handling call
0905 19:59:57.866416 (+ 84521us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:58.460884 (+594468us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:58.506093 (+ 45209us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:59.032980 28076 rpcz_store.cc:276] 0905 19:59:57.823846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:57.938742 (+114896us) service_pool.cc:225] Handling call
0905 19:59:58.973483 (+1034741us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:59.032006 30262 raft_consensus.cc:695] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:59.094661 27973 leader_election.cc:304] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 19:59:59.038126 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:59.095819 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:59.043641 28154 raft_consensus.cc:2466] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6.
I20250905 19:59:59.100581 30106 leader_election.cc:290] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:59.101507 28155 raft_consensus.cc:1238] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:58.680954 30383 consensus_queue.cc:260] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:59.081593 30017 leader_election.cc:290] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:59.184577 30419 raft_consensus.cc:491] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:59.193801 30419 raft_consensus.cc:513] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:59.194571 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 19:59:59.162019 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 19:59:59.196307 28008 raft_consensus.cc:2408] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250905 19:59:59.133155 28003 rpcz_store.cc:276] 0905 19:59:57.909404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:59.028550 (+1119146us) service_pool.cc:225] Handling call
0905 19:59:59.132527 (+103977us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:59.167084 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 19:59:59.240765 28001 raft_consensus.cc:2391] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 6: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 19:59:59.124863 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:59.273913 28001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1405) took 1255 ms. Trace:
I20250905 19:59:59.274752 28001 rpcz_store.cc:276] 0905 19:59:58.019060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:59.161549 (+1142489us) service_pool.cc:225] Handling call
0905 19:59:59.273732 (+112183us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:59.245209 28010 raft_consensus.cc:2466] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 19:59:59.283706 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:59.288162 28155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39586 (request call id 271) took 1555 ms. Trace:
I20250905 19:59:59.288885 28155 rpcz_store.cc:276] 0905 19:59:57.733227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:57.763312 (+ 30085us) service_pool.cc:225] Handling call
0905 19:59:57.785989 (+ 22677us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:59.286811 (+1500822us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:59.287747 (+ 936us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b540026cad4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:59.288059 (+ 312us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3024317184}
I20250905 19:59:59.290781 28009 raft_consensus.cc:3058] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:59.242401 30419 leader_election.cc:290] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:59.294628 30419 raft_consensus.cc:491] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 19:59:59.296770 30262 consensus_queue.cc:237] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:59.300899 28151 raft_consensus.cc:1238] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:59.301761 28156 raft_consensus.cc:2466] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 7.
I20250905 19:59:59.302886 28151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2070) took 1633 ms. Trace:
I20250905 19:59:59.303431 28151 rpcz_store.cc:276] 0905 19:59:57.670033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:57.671647 (+ 1614us) service_pool.cc:225] Handling call
0905 19:59:57.671947 (+ 300us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:59.301587 (+1629640us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:59.302510 (+ 923us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b5400201054. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 19:59:59.302800 (+ 290us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3745112448}
I20250905 19:59:59.305193 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:59.307569 30335 leader_election.cc:290] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 19:59:59.310243 28154 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39586 (request call id 272) took 1576 ms (client timeout 1774 ms). Trace:
W20250905 19:59:59.310938 28154 rpcz_store.cc:269] 0905 19:59:57.733520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:57.734683 (+ 1163us) service_pool.cc:225] Handling call
0905 19:59:59.309914 (+1575231us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54000e1ad4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:59.310126 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2556168576}
I20250905 19:59:59.196141 30383 raft_consensus.cc:3058] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:59:59.312690 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:59.313861 28154 raft_consensus.cc:2408] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250905 19:59:59.315075 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:59.315869 28077 raft_consensus.cc:1273] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250905 19:59:59.316306 28153 raft_consensus.cc:2466] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 7.
I20250905 19:59:59.316814 28077 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1472) took 1474 ms. Trace:
I20250905 19:59:59.241044 28008 raft_consensus.cc:1238] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20250905 19:59:59.319880 28151 raft_consensus.cc:2391] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 4: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:59.286015 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 19:59:59.320876 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:59.317873 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:59.150096 28004 rpcz_store.cc:276] 0905 19:59:57.850258 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:59.010244 (+1159986us) service_pool.cc:225] Handling call
0905 19:59:59.149000 (+138756us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b540027f954. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 19:59:59.149257 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":312525184}
I20250905 19:59:59.278291 28010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1394) took 1427 ms. Trace:
W20250905 19:59:59.330339 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 248076us
I20250905 19:59:59.330634 30017 raft_consensus.cc:2802] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:59:59.330420 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 19:59:59.317392 28077 rpcz_store.cc:276] 0905 19:59:57.842471 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:59.314725 (+1472254us) service_pool.cc:225] Handling call
0905 19:59:59.315256 (+ 531us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:59.316430 (+ 1174us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:59.316710 (+ 280us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:59.154305 28051 leader_election.cc:336] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.862s (SENT)
I20250905 19:59:59.336544 28748 raft_consensus.cc:1273] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250905 19:59:59.296797 30419 raft_consensus.cc:513] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:59.338793 28155 raft_consensus.cc:1238] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 2. Current term is 3. Ops: []
I20250905 19:59:59.339082 28156 raft_consensus.cc:1238] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20250905 19:59:59.372807 30375 consensus_queue.cc:1046] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 19:59:59.374142 30375 raft_consensus.cc:3053] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 19:59:59.374622 30375 raft_consensus.cc:738] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 19:59:59.339696 28153 raft_consensus.cc:2391] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 4: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 19:59:59.333982 30017 raft_consensus.cc:695] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 19:59:59.377445 30017 consensus_queue.cc:237] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:59.377522 30375 consensus_queue.cc:260] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 19:59:59.347888 30406 leader_election.cc:290] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:59.362394 30451 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:59.331378 28008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1386) took 1479 ms. Trace:
I20250905 19:59:59.389371 28008 rpcz_store.cc:276] 0905 19:59:57.840169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:59.239773 (+1399604us) service_pool.cc:225] Handling call
0905 19:59:59.240549 (+ 776us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:59.319456 (+ 78907us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:59.319752 (+ 296us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:59.388012 28154 raft_consensus.cc:1238] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20250905 19:59:59.114030 30371 leader_election.cc:290] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:59.243919 30420 raft_consensus.cc:3058] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 19:59:59.329332 28010 rpcz_store.cc:276] 0905 19:59:57.851145 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:58.773961 (+922816us) service_pool.cc:225] Handling call
0905 19:59:59.278203 (+504242us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:59.240262 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 19:59:59.410380 28006 raft_consensus.cc:2408] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20250905 19:59:59.403927 30375 raft_consensus.cc:3058] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
W20250905 19:59:59.743685 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:59:59.609994 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 80022.6us
I20250905 19:59:59.446905 28003 raft_consensus.cc:1273] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250905 19:59:59.449643 28004 raft_consensus.cc:1238] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 3. Current term is 4. Ops: []
I20250905 19:59:59.481565 28006 raft_consensus.cc:1273] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250905 19:59:59.653815 28005 raft_consensus.cc:2466] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
W20250905 19:59:59.457083 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:59.485857 30419 leader_election.cc:290] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 19:59:59.749099 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 19:59:59.468257 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:59.794999 28003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1387) took 1951 ms. Trace:
I20250905 19:59:59.795544 28003 rpcz_store.cc:276] 0905 19:59:57.840458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:59.335239 (+1494781us) service_pool.cc:225] Handling call
0905 19:59:59.446389 (+111150us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:59.791246 (+344857us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:59.791499 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:59.479218 30381 consensus_meta.cc:287] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.503s user 0.014s sys 0.000s
I20250905 19:59:59.740625 28001 raft_consensus.cc:1273] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250905 19:59:59.740461 28008 raft_consensus.cc:1238] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 7. Current term is 8. Ops: []
W20250905 19:59:59.749145 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 594874us
I20250905 19:59:59.492652 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:59.729300 30451 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:59.775040 28152 raft_consensus.cc:2466] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 19:59:59.720036 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 19:59:59.783336 30232 leader_election.cc:290] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 19:59:59.787288 28009 raft_consensus.cc:2408] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250905 19:59:59.753986 28007 raft_consensus.cc:2466] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 19:59:59.798337 28006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1399) took 1883 ms. Trace:
I20250905 19:59:59.808857 28009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 2009) took 1043 ms. Trace:
I20250905 19:59:59.809012 28010 raft_consensus.cc:1238] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: []
I20250905 19:59:59.809314 28006 rpcz_store.cc:276] 0905 19:59:57.909088 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:59.448452 (+1539364us) service_pool.cc:225] Handling call
0905 19:59:59.470870 (+ 22418us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:59.792469 (+321599us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:59.792712 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:59.802366 28008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1396) took 1914 ms. Trace:
I20250905 19:59:59.811369 28008 rpcz_store.cc:276] 0905 19:59:57.888020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:59.404210 (+1516190us) service_pool.cc:225] Handling call
0905 19:59:59.466251 (+ 62041us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:59.800610 (+334359us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:59.802303 (+ 1693us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:59.809753 28009 rpcz_store.cc:276] 0905 19:59:58.765864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:59.044405 (+278541us) service_pool.cc:225] Handling call
0905 19:59:59.808752 (+764347us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:59.804129 28051 leader_election.cc:336] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.880s (SENT)
I20250905 19:59:59.800742 28001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1388) took 1960 ms. Trace:
I20250905 19:59:59.815012 28001 rpcz_store.cc:276] 0905 19:59:57.845565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:59.394467 (+1548902us) service_pool.cc:225] Handling call
0905 19:59:59.452139 (+ 57672us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:59.800337 (+348198us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:59.800665 (+ 328us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:59.799299 28004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1390) took 1945 ms. Trace:
I20250905 19:59:59.816319 28004 rpcz_store.cc:276] 0905 19:59:57.846485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:59.335332 (+1488847us) service_pool.cc:225] Handling call
0905 19:59:59.425121 (+ 89789us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:59.791910 (+366789us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:59.792143 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:59.819572 28006 raft_consensus.cc:1238] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250905 19:59:59.820721 28006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 2002) took 1894 ms. Trace:
W20250905 19:59:59.821684 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:59.823314 28006 rpcz_store.cc:276] 0905 19:59:57.926008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:59.818735 (+1892727us) service_pool.cc:225] Handling call
0905 19:59:59.819226 (+ 491us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:59.820320 (+ 1094us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:59.820604 (+ 284us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:59.825675 30232 raft_consensus.cc:491] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:59.710911 29941 raft_consensus.cc:491] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:59:59.650916 30148 consensus_queue.cc:1046] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.163s
W20250905 19:59:59.871446 28746 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.455s user 0.010s sys 0.032s
W20250905 19:59:59.875511 28718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.320s user 0.017s sys 0.025s
W20250905 19:59:59.877084 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 490043us
I20250905 19:59:59.879827 28051 leader_election.cc:304] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 19:59:59.918004 28085 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.888s user 0.038s sys 0.088s
I20250905 19:59:59.881680 28009 raft_consensus.cc:1238] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
W20250905 19:59:59.893801 28158 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.030s user 0.008s sys 0.008s
W20250905 19:59:59.921000 28158 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.058s user 0.009s sys 0.008s
W20250905 19:59:59.923949 28718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.369s user 0.018s sys 0.026s
I20250905 19:59:59.918965 28009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 2003) took 1978 ms. Trace:
W20250905 19:59:59.927686 28085 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.962s user 0.038s sys 0.089s
I20250905 19:59:59.919014 28003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1400) took 1993 ms. Trace:
I20250905 19:59:59.927330 28009 rpcz_store.cc:276] 0905 19:59:57.940480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:59.853357 (+1912877us) service_pool.cc:225] Handling call
0905 19:59:59.881358 (+ 28001us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:59.918672 (+ 37314us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:59.918909 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:59:59.928577 28003 rpcz_store.cc:276] 0905 19:59:57.925323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:59.803934 (+1878611us) service_pool.cc:225] Handling call
0905 19:59:59.813283 (+ 9349us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:59.914156 (+100873us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:59.914207 (+ 51us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:59.918771 (+ 4564us) raft_consensus.cc:1643] Marking committed up to 3
0905 19:59:59.918821 (+ 50us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:59.918879 (+ 58us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:59.918968 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:59.935770 28079 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.950s user 0.007s sys 0.023s
W20250905 19:59:59.936452 28079 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.951s user 0.008s sys 0.023s
I20250905 19:59:59.937002 28158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2080) took 1136 ms. Trace:
W20250905 19:59:59.933665 28746 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.517s user 0.011s sys 0.032s
I20250905 19:59:59.951383 28009 raft_consensus.cc:1238] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 3. Current term is 4. Ops: []
W20250905 19:59:59.942791 28080 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.022s user 0.000s sys 0.019s
W20250905 19:59:59.953037 28080 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.032s user 0.000s sys 0.019s
I20250905 19:59:59.938091 30292 raft_consensus.cc:2747] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250905 19:59:59.896167 28123 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.372s user 0.001s sys 0.068s
I20250905 19:59:59.913563 30232 raft_consensus.cc:513] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 19:59:59.966624 28009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1406) took 1874 ms. Trace:
I20250905 19:59:59.967454 28009 rpcz_store.cc:276] 0905 19:59:58.080151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:59.929832 (+1849681us) service_pool.cc:225] Handling call
0905 19:59:59.951188 (+ 21356us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:59.951904 (+ 716us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:59.954380 (+ 2476us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8576}
W20250905 19:59:59.972244 28123 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.453s user 0.002s sys 0.069s
W20250905 19:59:59.959419 30017 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.577s user 0.001s sys 0.024s
I20250905 19:59:59.975421 28008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1401) took 2033 ms. Trace:
W20250905 19:59:59.980041 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:59:59.977890 28008 rpcz_store.cc:276] 0905 19:59:57.937326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:59.818481 (+1881155us) service_pool.cc:225] Handling call
0905 19:59:59.936524 (+118043us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:59.969884 (+ 33360us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:59.969925 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:59.969979 (+ 54us) raft_consensus.cc:1643] Marking committed up to 2
0905 19:59:59.970035 (+ 56us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:59.970127 (+ 92us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:59.970311 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:59:59.977984 28715 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.894s user 0.050s sys 0.125s
W20250905 19:59:59.989521 28715 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.065s user 0.051s sys 0.125s
I20250905 19:59:59.927182 30383 consensus_queue.cc:1035] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.053s
W20250905 19:59:59.977819 30017 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.596s user 0.002s sys 0.024s
W20250905 19:59:59.894863 30393 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.182s user 0.002s sys 0.020s
W20250905 19:59:59.993911 28155 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.607s user 0.000s sys 0.012s
W20250905 20:00:00.003368 28155 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.629s user 0.002s sys 0.012s
W20250905 19:59:59.986153 28084 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.747s user 0.038s sys 0.046s
I20250905 19:59:59.978796 30148 raft_consensus.cc:3053] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 LEADER]: Stepping down as leader of term 8
W20250905 19:59:59.994367 28150 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.994s user 0.004s sys 0.011s
W20250905 20:00:00.009039 28081 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.423s user 0.027s sys 0.024s
W20250905 20:00:00.009781 30442 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.397s user 0.003s sys 0.021s
W20250905 20:00:00.011719 28150 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.011s user 0.005s sys 0.011s
W20250905 19:59:59.995751 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 1.47786e+06us
W20250905 20:00:00.009573 30240 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.922s user 0.012s sys 0.037s
W20250905 19:59:59.992470 28125 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.710s (SENT) [suppressed 15 similar messages]
I20250905 19:59:59.961301 29941 raft_consensus.cc:513] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:00.023927 28081 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.524s user 0.027s sys 0.025s
W20250905 20:00:00.010216 30372 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.095s user 0.007s sys 0.008s
W20250905 20:00:00.034746 28084 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.923s user 0.038s sys 0.047s
W20250905 20:00:00.037021 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 45699us
W20250905 20:00:00.018823 30393 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.298s user 0.003s sys 0.021s
W20250905 20:00:00.041026 28083 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.111s user 0.022s sys 0.055s
W20250905 20:00:00.044525 28083 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.115s user 0.022s sys 0.055s
W20250905 20:00:00.034129 30240 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.964s user 0.014s sys 0.043s
W20250905 20:00:00.042800 27974 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.965s user 0.001s sys 0.027s
W20250905 20:00:00.051373 27974 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.085s user 0.001s sys 0.027s
W20250905 20:00:00.051903 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 1.08601e+06us
W20250905 20:00:00.040084 30372 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.141s user 0.008s sys 0.008s
W20250905 20:00:00.055460 28078 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.811s user 0.027s sys 0.044s
I20250905 20:00:00.010778 30148 raft_consensus.cc:738] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
W20250905 20:00:00.060199 28078 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.815s user 0.027s sys 0.044s
W20250905 20:00:00.045348 30442 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.433s user 0.003s sys 0.022s
I20250905 20:00:00.062191 28157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39586 (request call id 278) took 1001 ms. Trace:
W20250905 20:00:00.062181 28122 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.379s user 0.004s sys 0.013s
I20250905 20:00:00.068413 28717 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
W20250905 20:00:00.074918 28122 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.507s user 0.004s sys 0.014s
W20250905 20:00:00.076682 30433 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.405s user 0.017s sys 0.006s
W20250905 20:00:00.076953 28082 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.477s user 0.005s sys 0.013s
W20250905 20:00:00.077250 30433 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.444s user 0.018s sys 0.007s
W20250905 20:00:00.077528 28082 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.584s user 0.006s sys 0.013s
W20250905 20:00:00.081890 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 126838us
W20250905 20:00:00.083037 30263 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.316s user 0.035s sys 0.061s
W20250905 20:00:00.083508 30263 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.362s user 0.035s sys 0.061s
I20250905 20:00:00.083611 30450 raft_consensus.cc:2802] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 20:00:00.063563 28157 rpcz_store.cc:276] 0905 19:59:59.057163 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:59.333236 (+276073us) service_pool.cc:225] Handling call
0905 19:59:59.425371 (+ 92135us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:59.490640 (+ 65269us) raft_consensus.cc:1532] Early marking committed up to index 5
0905 19:59:59.625501 (+134861us) log.cc:844] Serialized 14 byte log entry
0905 20:00:00.050753 (+425252us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:00.050834 (+ 81us) raft_consensus.cc:1643] Marking committed up to 5
0905 20:00:00.058072 (+ 7238us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:00.058165 (+ 93us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:00.058334 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":121589,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":65,"raft.queue_time_us":11119,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":5945,"thread_start_us":335789,"threads_started":1}
W20250905 20:00:00.087970 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:00.089605 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:00.080525 30280 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.275s user 0.003s sys 0.015s
W20250905 20:00:00.090054 27975 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.139s user 0.008s sys 0.007s
W20250905 20:00:00.095281 27975 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.293s user 0.008s sys 0.008s
I20250905 20:00:00.090471 28006 raft_consensus.cc:1238] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 3. Current term is 4. Ops: []
W20250905 20:00:00.099402 30453 log.cc:927] Time spent T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.156s user 0.000s sys 0.001s
W20250905 20:00:00.100106 30379 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.521s user 0.012s sys 0.026s
W20250905 20:00:00.100641 30379 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.550s user 0.012s sys 0.026s
I20250905 20:00:00.084125 30450 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:00.096738 28081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1452) took 2717 ms. Trace:
W20250905 20:00:00.106930 30280 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.408s user 0.003s sys 0.016s
W20250905 20:00:00.131371 28122 connection.cc:232] RPC call timeout handler was delayed by -1.2730308800000785s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20800000000000002s for a response
W20250905 20:00:00.132242 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:00.133487 30393 raft_consensus.cc:491] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:00.135878 28010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1397) took 2235 ms. Trace:
I20250905 20:00:00.137890 28078 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1470) took 2308 ms. Trace:
I20250905 20:00:00.138391 28010 rpcz_store.cc:276] 0905 19:59:57.900253 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:59.426564 (+1526311us) service_pool.cc:225] Handling call
0905 19:59:59.808719 (+382155us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:59.810144 (+ 1425us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:00.135605 (+325461us) spinlock_profiling.cc:243] Waited 361 ms on lock 0x7b5400384ad0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:00.135801 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":829191424}
I20250905 20:00:00.106577 30263 raft_consensus.cc:491] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:00.139140 28078 rpcz_store.cc:276] 0905 19:59:57.829302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:59.138216 (+1308914us) service_pool.cc:225] Handling call
0905 19:59:59.242987 (+104771us) raft_consensus.cc:1469] Updating replica for 2 ops
0905 19:59:59.243897 (+ 910us) raft_consensus.cc:1532] Early marking committed up to index 2
0905 19:59:59.244363 (+ 466us) log.cc:844] Serialized 14 byte log entry
0905 20:00:00.060870 (+816507us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0905 20:00:00.069561 (+ 8691us) log.cc:844] Serialized 52 byte log entry
0905 20:00:00.069775 (+ 214us) raft_consensus.cc:1643] Marking committed up to 2
0905 20:00:00.069810 (+ 35us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:00.069886 (+ 76us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:00.137640 (+ 67754us) raft_consensus.cc:1683] finished
0905 20:00:00.137664 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:00.137825 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5581,"prepare.queue_time_us":699600,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":25148,"raft.run_cpu_time_us":294,"raft.run_wall_time_us":3982,"thread_start_us":815860,"threads_started":1,"wal-append.queue_time_us":889723}
I20250905 20:00:00.142110 30393 raft_consensus.cc:513] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:00.145504 30455 log.cc:927] Time spent T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.099s user 0.000s sys 0.001s
I20250905 20:00:00.147706 30393 leader_election.cc:290] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 20:00:00.141745 30346 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.575s user 0.010s sys 0.023s
W20250905 20:00:00.148900 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:00.149039 30346 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.682s user 0.011s sys 0.023s
W20250905 20:00:00.149746 28005 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1382) took 2316 ms (client timeout 1538 ms). Trace:
I20250905 20:00:00.149767 30468 raft_consensus.cc:491] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:00.109829 30450 raft_consensus.cc:3058] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:00.110216 30240 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:33742 (request call id 673) took 5437 ms. Trace:
I20250905 20:00:00.157390 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:00.152256 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:00.150287 28005 rpcz_store.cc:269] 0905 19:59:57.832930 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:57.849152 (+ 16222us) service_pool.cc:225] Handling call
0905 20:00:00.142748 (+2293596us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54001fb654. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:00.149576 (+ 6828us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2555351296}
I20250905 20:00:00.158676 28157 raft_consensus.cc:2408] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 5 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250905 20:00:00.153673 30232 leader_election.cc:290] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:00.107689 28081 rpcz_store.cc:276] 0905 19:59:57.370627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:57.786461 (+415834us) service_pool.cc:225] Handling call
0905 19:59:57.786878 (+ 417us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:57.787350 (+ 472us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:57.787394 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:57.822454 (+ 35060us) log.cc:844] Serialized 26 byte log entry
0905 19:59:58.034709 (+212255us) raft_consensus.cc:1643] Marking committed up to 3
0905 20:00:00.039572 (+2004863us) log.cc:844] Serialized 14 byte log entry
0905 20:00:00.066957 (+ 27385us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:00.081567 (+ 14610us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:00.081622 (+ 55us) raft_consensus.cc:1683] finished
0905 20:00:00.081634 (+ 12us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:00.087925 (+ 6291us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":457995,"prepare.queue_time_us":549794,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":10091,"raft.run_cpu_time_us":404,"raft.run_wall_time_us":44245,"thread_start_us":1663178,"threads_started":2,"wal-append.queue_time_us":1886222}
I20250905 20:00:00.156749 29941 leader_election.cc:290] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:00.157013 28153 raft_consensus.cc:1273] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250905 20:00:00.165529 30263 raft_consensus.cc:513] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:00.166867 28084 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1465) took 2642 ms. Trace:
W20250905 20:00:00.170946 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 43705.7us
W20250905 20:00:00.173151 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:00.174582 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:00.167626 28745 raft_consensus.cc:1273] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20250905 20:00:00.182667 30263 leader_election.cc:290] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:00.183734 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:00.150698 30468 raft_consensus.cc:513] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:00.159130 30319 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.376s user 0.006s sys 0.035s
I20250905 20:00:00.180686 28080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 250) took 2484 ms. Trace:
I20250905 20:00:00.191190 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:00.172109 28084 rpcz_store.cc:276] 0905 19:59:57.524559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:57.950955 (+426396us) service_pool.cc:225] Handling call
0905 19:59:57.974803 (+ 23848us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:57.987074 (+ 12271us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:57.987108 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:58.001231 (+ 14123us) log.cc:844] Serialized 26 byte log entry
0905 20:00:00.087423 (+2086192us) raft_consensus.cc:1643] Marking committed up to 3
0905 20:00:00.090480 (+ 3057us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:00.106755 (+ 16275us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:00.150869 (+ 44114us) raft_consensus.cc:1683] finished
0905 20:00:00.150891 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:00.166790 (+ 15899us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4029,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":15093,"thread_start_us":1929117,"threads_started":1,"wal-append.queue_time_us":1925422}
I20250905 20:00:00.180254 28745 pending_rounds.cc:85] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a: Aborting all ops after (but not including) 6
I20250905 20:00:00.206161 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:00.178100 30393 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:00.192258 28080 rpcz_store.cc:276] 0905 19:59:57.691711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:58.911580 (+1219869us) service_pool.cc:225] Handling call
0905 19:59:58.912586 (+ 1006us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:58.919346 (+ 6760us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:58.919402 (+ 56us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:58.920201 (+ 799us) log.cc:844] Serialized 26 byte log entry
0905 19:59:59.953611 (+1033410us) raft_consensus.cc:1643] Marking committed up to 6
0905 19:59:59.986082 (+ 32471us) log.cc:844] Serialized 14 byte log entry
0905 19:59:59.990157 (+ 4075us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:59.990257 (+ 100us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:00.077695 (+ 87438us) spinlock_profiling.cc:243] Waited 5.56 ms on lock 0x7b180042aa80. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49aca43e2 00007fe49aca67b1 00007fe49aca6769 00007fe49aca66f9 00007fe49aca66c1 00007fe49aca59bd 0000000000379254 0000000000379099 00007fe49ac8776a
0905 20:00:00.168285 (+ 90590us) raft_consensus.cc:1683] finished
0905 20:00:00.175624 (+ 7339us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:00.175807 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":36651,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":48,"raft.queue_time_us":11524,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":270,"spinlock_wait_cycles":23839232,"thread_start_us":1032866,"threads_started":1,"wal-append.queue_time_us":1244908}
W20250905 20:00:00.206321 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:00.208921 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:00.210268 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:00.215562 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:00.188846 30419 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:00.216775 28156 raft_consensus.cc:2466] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
W20250905 20:00:00.216670 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 44260.9us
W20250905 20:00:00.198637 30319 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.528s user 0.007s sys 0.036s
I20250905 20:00:00.217962 28154 raft_consensus.cc:2466] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 20:00:00.221369 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:00.218585 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:00.223301 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:00.227749 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:00.228940 28156 raft_consensus.cc:2391] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 4: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:00.224349 28157 raft_consensus.cc:2466] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 20:00:00.230404 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:00.204216 28745 pending_rounds.cc:107] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20250905 20:00:00.241256 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:00.222093 28081 raft_consensus.cc:2466] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:00.206719 28077 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Advancing to term 7
W20250905 20:00:00.252990 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:00.254978 30459 raft_consensus.cc:491] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:00.253643 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:00.240017 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:00.262017 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:00.265017 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 44087.4us
I20250905 20:00:00.182158 30458 consensus_queue.cc:1046] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20250905 20:00:00.182117 30148 consensus_queue.cc:260] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:00.273142 28078 raft_consensus.cc:3058] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:00.254552 28745 raft_consensus.cc:2887] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250905 20:00:00.278728 28159 raft_consensus.cc:2408] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250905 20:00:00.275000 28123 leader_election.cc:304] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:00.281482 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:00.269446 30459 raft_consensus.cc:513] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:00.285229 28751 raft_consensus.cc:1238] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 6. Current term is 7. Ops: []
W20250905 20:00:00.276714 30469 log.cc:927] Time spent T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.090s user 0.001s sys 0.000s
I20250905 20:00:00.289623 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:00.299315 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
W20250905 20:00:00.300350 28122 connection.cc:232] RPC call timeout handler was delayed by -1.049626676999992s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.171s for a response
I20250905 20:00:00.157991 30240 rpcz_store.cc:276] 0905 19:59:54.668533 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:55.037670 (+369137us) service_pool.cc:225] Handling call
0905 19:59:55.188931 (+151261us) log.cc:844] Serialized 501 byte log entry
0905 19:59:55.923959 (+735028us) spinlock_profiling.cc:243] Waited 22.9 ms on lock 0x7b18003ebcc0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49aca43e2 00007fe49aca67b1 00007fe49aca6769 00007fe49aca66f9 00007fe49aca66c1 00007fe49aca59bd 0000000000379254 0000000000379099 00007fe49ac8776a
0905 19:59:56.601076 (+677117us) spinlock_profiling.cc:243] Waited 676 ms on lock 0x7b580036cbb0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9dcd0 00007fe49db9288c 00007fe49db97911 00007fe49db97899 00007fe49db97829 00007fe49db977f1 00007fe49db969bd 0000000000379254 0000000000379099 00007fe49aca4464 00007fe49aca67b1
0905 20:00:00.092217 (+3491141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9097,"raft.queue_time_us":28407,"raft.run_cpu_time_us":3850,"raft.run_wall_time_us":677703,"spinlock_wait_cycles":1608476544}
I20250905 20:00:00.306828 30471 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:00.307535 30471 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:00.308507 30468 leader_election.cc:290] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:00.309566 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:00.309901 28010 raft_consensus.cc:1238] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:00.270753 30393 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:00.304623 28747 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Advancing to term 9
I20250905 20:00:00.313395 30346 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:00.371803 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:00.305392 30460 log.cc:927] Time spent T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.100s user 0.000s sys 0.001s
I20250905 20:00:00.391685 30319 consensus_queue.cc:1035] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250905 20:00:00.415737 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:00.303110 30420 raft_consensus.cc:2762] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20250905 20:00:00.431139 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:00.429909 27975 leader_election.cc:304] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:00.297008 30419 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:00.445156 28153 raft_consensus.cc:3058] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:00.415575 30458 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
W20250905 20:00:00.448355 30486 log.cc:927] Time spent T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.127s user 0.000s sys 0.000s
I20250905 20:00:00.448262 30458 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:00.448124 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:00.450146 28751 raft_consensus.cc:2408] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20250905 20:00:00.450641 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:00.450613 28159 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
from {username='slave'} at 127.0.0.1:36164
I20250905 20:00:00.451681 28159 raft_consensus.cc:604] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Received request to transfer leadership to TS 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:00.452136 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:00.452390 28007 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1398) took 2550 ms (client timeout 1710 ms). Trace:
W20250905 20:00:00.452814 28007 rpcz_store.cc:269] 0905 19:59:57.902264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:57.934305 (+ 32041us) service_pool.cc:225] Handling call
0905 20:00:00.452168 (+2517863us) spinlock_profiling.cc:243] Waited 1.74 s on lock 0x7b54001324d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:00.452317 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3998777728}
W20250905 20:00:00.453161 28152 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33752 (request call id 1519) took 1604 ms (client timeout 1867 ms). Trace:
W20250905 20:00:00.453651 28152 rpcz_store.cc:269] 0905 19:59:58.848962 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:58.920069 (+ 71107us) service_pool.cc:225] Handling call
0905 20:00:00.452991 (+1532922us) spinlock_profiling.cc:243] Waited 789 ms on lock 0x7b5400176e54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:00.453096 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1814265216}
I20250905 20:00:00.450119 28004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 2004) took 2485 ms. Trace:
I20250905 20:00:00.454182 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:00.454607 28004 rpcz_store.cc:276] 0905 19:59:57.967502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:59.859077 (+1891575us) service_pool.cc:225] Handling call
0905 19:59:59.886810 (+ 27733us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:59.923171 (+ 36361us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:59.923220 (+ 49us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:00.104377 (+181157us) spinlock_profiling.cc:243] Waited 1.96 ms on lock 0x7b2c000014c8. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49c5bc9b1 00007fe49c5cbfe2 00007fe49dc55021 00007fe49dc193fa 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32
0905 20:00:00.126599 (+ 22222us) log.cc:844] Serialized 26 byte log entry
0905 20:00:00.196737 (+ 70138us) raft_consensus.cc:1643] Marking committed up to 1
0905 20:00:00.196784 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:00.196874 (+ 90us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:00.449826 (+252952us) raft_consensus.cc:1683] finished
0905 20:00:00.449848 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:00.450054 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4507904,"thread_start_us":65862,"threads_started":1,"wal-append.queue_time_us":157613}
I20250905 20:00:00.412191 30383 raft_consensus.cc:491] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:00.457687 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:00.303313 30381 consensus_queue.cc:1046] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.133s
I20250905 20:00:00.452122 28159 raft_consensus.cc:612] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250905 20:00:00.464448 28747 raft_consensus.cc:2408] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20250905 20:00:00.444836 28051 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.005s sys 0.004s
W20250905 20:00:00.470042 28051 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.552s user 0.005s sys 0.005s
I20250905 20:00:00.358805 28746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40904 (request call id 566) took 3365 ms. Trace:
I20250905 20:00:00.467578 28078 raft_consensus.cc:2466] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
I20250905 20:00:00.430898 30379 raft_consensus.cc:491] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:00.480871 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:00.321595 30262 consensus_queue.cc:1035] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.142s
W20250905 20:00:00.488756 28151 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.022s sys 0.011s
W20250905 20:00:00.491097 28151 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.578s user 0.023s sys 0.011s
I20250905 20:00:00.492918 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:00.488359 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:00.481928 28122 leader_election.cc:336] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.538s (SENT)
I20250905 20:00:00.500990 28010 raft_consensus.cc:2466] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 20:00:00.504175 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:00.417011 30346 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:00.511792 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:00.512542 28078 raft_consensus.cc:2391] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 4: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:00.520994 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:00.497954 30263 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:00.523363 28150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1514) took 1677 ms. Trace:
W20250905 20:00:00.523923 28122 leader_election.cc:336] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.613s (SENT)
I20250905 20:00:00.526355 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:00.526562 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 722455us
I20250905 20:00:00.524923 28081 raft_consensus.cc:1238] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 2. Current term is 3. Ops: []
I20250905 20:00:00.502635 30459 leader_election.cc:290] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:00.489719 30383 raft_consensus.cc:513] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:00.525782 30471 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:00.554488 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:00.535913 30319 raft_consensus.cc:2747] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:00.566524 29941 raft_consensus.cc:491] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:00.569650 28010 raft_consensus.cc:1273] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250905 20:00:00.570595 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 20:00:00.571635 30461 log.cc:927] Time spent T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.099s user 0.000s sys 0.001s
I20250905 20:00:00.536404 28122 leader_election.cc:304] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:00.574208 28153 raft_consensus.cc:2466] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 7.
W20250905 20:00:00.577826 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:00.579807 30346 leader_election.cc:290] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:00.494418 30379 raft_consensus.cc:513] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:00.580139 30458 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:00.580835 30419 leader_election.cc:290] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:00.581233 28002 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1385) took 2741 ms (client timeout 1734 ms). Trace:
I20250905 20:00:00.530972 30263 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 19:59:59.939263 28158 rpcz_store.cc:276] 0905 19:59:58.800529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:58.860306 (+ 59777us) service_pool.cc:225] Handling call
0905 19:59:58.861101 (+ 795us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:58.861595 (+ 494us) raft_consensus.cc:1532] Early marking committed up to index 3
0905 19:59:58.862249 (+ 654us) log.cc:844] Serialized 14 byte log entry
0905 19:59:59.931649 (+1069400us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 19:59:59.931738 (+ 89us) raft_consensus.cc:1643] Marking committed up to 3
0905 19:59:59.931831 (+ 93us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:59.931933 (+ 102us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:59:59.936923 (+ 4990us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":4895,"prepare.run_cpu_time_us":136,"prepare.run_wall_time_us":31303,"raft.queue_time_us":16475,"thread_start_us":1058615,"threads_started":1}
W20250905 20:00:00.598063 28002 rpcz_store.cc:269] 0905 19:59:57.839815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:57.842891 (+ 3076us) service_pool.cc:225] Handling call
0905 20:00:00.580807 (+2737916us) spinlock_profiling.cc:243] Waited 2.35 s on lock 0x7b540016dfd4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:00.581025 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5402374912}
I20250905 20:00:00.525061 28150 rpcz_store.cc:276] 0905 19:59:58.846197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:58.994799 (+148602us) service_pool.cc:225] Handling call
0905 19:59:58.999626 (+ 4827us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:59.000232 (+ 606us) raft_consensus.cc:1532] Early marking committed up to index 4
0905 20:00:00.012356 (+1012124us) log.cc:844] Serialized 14 byte log entry
0905 20:00:00.522577 (+510221us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:00.522707 (+ 130us) raft_consensus.cc:1643] Marking committed up to 4
0905 20:00:00.522873 (+ 166us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:00.523176 (+ 303us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:00.523299 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":539,"prepare.queue_time_us":28042,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":53,"raft.queue_time_us":6901,"thread_start_us":1463273,"threads_started":2}
W20250905 20:00:00.588328 28051 leader_election.cc:336] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.882s (SENT)
W20250905 20:00:00.572669 30474 log.cc:927] Time spent T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.152s user 0.000s sys 0.001s
I20250905 20:00:00.593523 30488 raft_consensus.cc:491] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:00.594846 30473 log.cc:927] Time spent T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.178s user 0.001s sys 0.000s
I20250905 20:00:00.605893 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:00.606643 30379 leader_election.cc:290] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:00.600589 28085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1443) took 3633 ms. Trace:
I20250905 20:00:00.609063 28051 leader_election.cc:304] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 20:00:00.600980 30375 consensus_meta.cc:287] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.809s user 0.010s sys 0.006s
I20250905 20:00:00.613358 28008 raft_consensus.cc:1273] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20250905 20:00:00.616720 30466 log.cc:927] Time spent T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.302s user 0.000s sys 0.001s
I20250905 20:00:00.617810 30383 leader_election.cc:290] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:00.618045 28002 raft_consensus.cc:1273] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 20:00:00.618317 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:00.607239 30240 consensus_queue.cc:1046] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.012s
I20250905 20:00:00.619508 28084 raft_consensus.cc:2408] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250905 20:00:00.620103 30240 raft_consensus.cc:3053] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Stepping down as leader of term 4
I20250905 20:00:00.620513 30240 raft_consensus.cc:738] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:00.621624 30240 consensus_queue.cc:260] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:00.615415 28125 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:00.624672 28081 raft_consensus.cc:2408] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250905 20:00:00.597445 30381 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:00.596719 30393 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:00.606710 30459 raft_consensus.cc:2747] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 20:00:00.623047 30240 raft_consensus.cc:3058] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
W20250905 20:00:00.663679 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 146414us
W20250905 20:00:00.668112 30480 log.cc:927] Time spent T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.402s user 0.000s sys 0.004s
I20250905 20:00:00.620118 30379 consensus_queue.cc:1046] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
W20250905 20:00:00.668146 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 57920.6us
I20250905 20:00:00.672513 28155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2085) took 1468 ms. Trace:
I20250905 20:00:00.667577 30381 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:00.672897 30495 raft_consensus.cc:491] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:00.609387 28085 rpcz_store.cc:276] 0905 19:59:56.966784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:57.617093 (+650309us) service_pool.cc:225] Handling call
0905 19:59:57.642061 (+ 24968us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:57.667008 (+ 24947us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:57.667046 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:57.687137 (+ 20091us) log.cc:844] Serialized 26 byte log entry
0905 19:59:57.930528 (+243391us) raft_consensus.cc:1643] Marking committed up to 2
0905 19:59:59.944801 (+2014273us) log.cc:844] Serialized 14 byte log entry
0905 20:00:00.513644 (+568843us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:00.523687 (+ 10043us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:00.586507 (+ 62820us) raft_consensus.cc:1683] finished
0905 20:00:00.586524 (+ 17us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:00.600519 (+ 13995us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21633,"prepare.queue_time_us":37977,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":31954,"raft.run_cpu_time_us":326,"raft.run_wall_time_us":15176,"thread_start_us":2667426,"threads_started":3,"wal-append.queue_time_us":857479,"wal-append.run_cpu_time_us":2958,"wal-append.run_wall_time_us":1401194}
I20250905 20:00:00.673000 28155 rpcz_store.cc:276] 0905 19:59:59.204479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:59.372303 (+167824us) service_pool.cc:225] Handling call
0905 19:59:59.372750 (+ 447us) raft_consensus.cc:1469] Updating replica for 2 ops
0905 19:59:59.373288 (+ 538us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:59.373316 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0905 19:59:59.374041 (+ 725us) log.cc:844] Serialized 52 byte log entry
0905 20:00:00.003925 (+629884us) raft_consensus.cc:1643] Marking committed up to 6
0905 20:00:00.009404 (+ 5479us) log.cc:844] Serialized 14 byte log entry
0905 20:00:00.009689 (+ 285us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:00.009783 (+ 94us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:00.157236 (+147453us) spinlock_profiling.cc:243] Waited 5.53 ms on lock 0x7b18003c30a0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49aca43e2 00007fe49aca67b1 00007fe49aca6769 00007fe49aca66f9 00007fe49aca66c1 00007fe49aca59bd 0000000000379254 0000000000379099 00007fe49ac8776a
0905 20:00:00.601820 (+444584us) spinlock_profiling.cc:243] Waited 1.37 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49d85891e 00007fe49d857c66 00007fe49d857888 00007fe49d8612c1 00007fe49d861279 00007fe49d861209
0905 20:00:00.672237 (+ 70417us) raft_consensus.cc:1683] finished
0905 20:00:00.672259 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:00.672444 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":15008,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":2767,"raft.run_cpu_time_us":468,"raft.run_wall_time_us":5881,"spinlock_wait_cycles":15881216,"thread_start_us":629364,"threads_started":1,"wal-append.queue_time_us":833552}
W20250905 20:00:00.675803 30450 consensus_meta.cc:287] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.510s user 0.009s sys 0.003s
I20250905 20:00:00.600879 29941 raft_consensus.cc:513] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:00.682109 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:00.684792 28081 raft_consensus.cc:2466] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 20:00:00.673578 30375 raft_consensus.cc:2747] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 20:00:00.667593 30419 raft_consensus.cc:2747] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:00.686460 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:00.688558 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 205618us
I20250905 20:00:00.705010 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 20:00:00.708456 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 297442us
I20250905 20:00:00.623649 30488 raft_consensus.cc:513] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:00.710036 28001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1402) took 2753 ms. Trace:
I20250905 20:00:00.709882 28125 leader_election.cc:304] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:00.712381 28079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 252) took 3020 ms. Trace:
I20250905 20:00:00.714147 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:00.713201 28079 rpcz_store.cc:276] 0905 19:59:57.692255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:58.929241 (+1236986us) service_pool.cc:225] Handling call
0905 19:59:58.929820 (+ 579us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:58.984175 (+ 54355us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:58.984224 (+ 49us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:58.984904 (+ 680us) log.cc:844] Serialized 26 byte log entry
0905 19:59:59.936974 (+952070us) raft_consensus.cc:1643] Marking committed up to 4
0905 19:59:59.937015 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:59:59.937092 (+ 77us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:00.712142 (+775050us) raft_consensus.cc:1683] finished
0905 20:00:00.712169 (+ 27us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:00.712314 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":200,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":225,"thread_start_us":951553,"threads_started":1,"wal-append.queue_time_us":1290238}
W20250905 20:00:00.710098 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 127340us
W20250905 20:00:00.719444 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 131168us
I20250905 20:00:00.687317 30450 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:00.703704 28076 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1471) took 2870 ms. Trace:
I20250905 20:00:00.733635 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:00.730711 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:00.683857 30495 raft_consensus.cc:513] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:00.714419 28752 raft_consensus.cc:2953] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 4 to 6, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 20:00:00.719164 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:00.727149 30419 raft_consensus.cc:2747] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250905 20:00:00.744267 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:00.756680 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 61660.9us
I20250905 20:00:00.747509 30496 raft_consensus.cc:3053] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 6 LEADER]: Stepping down as leader of term 6
I20250905 20:00:00.753490 30442 raft_consensus.cc:3053] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 2 LEADER]: Stepping down as leader of term 2
W20250905 20:00:00.753789 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:00.763063 27975 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:00.764433 30450 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:00.765774 30263 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:00.738487 28076 rpcz_store.cc:276] 0905 19:59:57.831530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:59.240722 (+1409192us) service_pool.cc:225] Handling call
0905 19:59:59.241474 (+ 752us) raft_consensus.cc:1469] Updating replica for 2 ops
0905 19:59:59.395498 (+154024us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:59.414082 (+ 18584us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0905 19:59:59.519145 (+105063us) log.cc:844] Serialized 52 byte log entry
0905 20:00:00.067865 (+548720us) raft_consensus.cc:1643] Marking committed up to 6
0905 20:00:00.098350 (+ 30485us) log.cc:844] Serialized 14 byte log entry
0905 20:00:00.666910 (+568560us) log.cc:844] Serialized 14 byte log entry
0905 20:00:00.678498 (+ 11588us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:00.684622 (+ 6124us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:00.689161 (+ 4539us) raft_consensus.cc:1683] finished
0905 20:00:00.689186 (+ 25us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:00.702190 (+ 13004us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19448,"prepare.queue_time_us":28340,"prepare.run_cpu_time_us":110,"prepare.run_wall_time_us":112,"raft.queue_time_us":190936,"raft.run_cpu_time_us":450,"raft.run_wall_time_us":5720,"spinlock_wait_cycles":30544256,"thread_start_us":814502,"threads_started":2,"wal-append.queue_time_us":487739}
I20250905 20:00:00.769726 27975 leader_election.cc:304] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:00.770936 30497 raft_consensus.cc:2747] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250905 20:00:00.771265 27975 leader_election.cc:304] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:00.771745 30497 raft_consensus.cc:2747] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 20:00:00.771826 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:00.772153 30232 raft_consensus.cc:2747] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250905 20:00:00.773176 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:00.773815 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:00.774158 27975 leader_election.cc:304] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
W20250905 20:00:00.775080 30433 log.cc:927] Time spent T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410: Append to log took a long time: real 0.089s user 0.001s sys 0.000s
I20250905 20:00:00.775179 30383 raft_consensus.cc:2747] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 20:00:00.477841 28746 rpcz_store.cc:276] 0905 19:59:56.953280 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:56.959056 (+ 5776us) service_pool.cc:225] Handling call
0905 19:59:57.247976 (+288920us) spinlock_profiling.cc:243] Waited 4.15 ms on lock 0x7b5800304598. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b91f60 00007fe4a2b7d1a9 00007fe4a2b7cc52 00007fe49c202b44
0905 19:59:57.668817 (+420841us) raft_consensus.cc:1469] Updating replica for 2 ops
0905 19:59:57.951992 (+283175us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:57.957972 (+ 5980us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0905 19:59:58.415783 (+457811us) log.cc:844] Serialized 52 byte log entry
0905 20:00:00.035599 (+1619816us) raft_consensus.cc:1643] Marking committed up to 6
0905 20:00:00.039200 (+ 3601us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:00.060537 (+ 21337us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:00.066819 (+ 6282us) raft_consensus.cc:1683] finished
0905 20:00:00.066835 (+ 16us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:00.302706 (+235871us) spinlock_profiling.cc:243] Waited 416 ms on lock 0x7b54003557d0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:00.318225 (+ 15519us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":65565,"raft.queue_time_us":254605,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":25296,"spinlock_wait_cycles":967170176,"thread_start_us":1719417,"threads_started":2,"wal-append.queue_time_us":1455541}
I20250905 20:00:00.775341 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:00.777519 27975 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:00.760174 30496 raft_consensus.cc:738] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:00.778682 30383 raft_consensus.cc:2802] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election won for term 6
I20250905 20:00:00.779621 30496 consensus_queue.cc:260] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:00.779732 30383 raft_consensus.cc:695] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
W20250905 20:00:00.780432 28077 consensus_meta.cc:287] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.510s user 0.006s sys 0.005s
I20250905 20:00:00.780864 28077 raft_consensus.cc:2466] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 7.
I20250905 20:00:00.781884 30383 consensus_queue.cc:237] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:00.783762 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:00.785051 30381 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:00.775106 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:00.781756 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:00.773290 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:00.791239 28008 raft_consensus.cc:2374] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4. Re-sending same reply.
I20250905 20:00:00.790220 28010 raft_consensus.cc:2374] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3. Re-sending same reply.
I20250905 20:00:00.798281 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:00.803005 28155 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
from {username='slave'} at 127.0.0.1:36164
I20250905 20:00:00.805119 28155 raft_consensus.cc:604] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Received request to transfer leadership to TS 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:00.805655 28002 raft_consensus.cc:2391] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 4: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 20:00:00.840962 28745 raft_consensus.cc:2391] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 7: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
W20250905 20:00:00.853210 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:00.817157 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:00.976033 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:00.976855 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:00.977396 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:00.805694 28155 raft_consensus.cc:612] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250905 20:00:00.878983 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:00.890904 30393 consensus_queue.cc:1035] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.075s
W20250905 20:00:00.917492 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 140068us
W20250905 20:00:00.980134 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:00.966508 30507 consensus_queue.cc:1035] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.179s
I20250905 20:00:00.980626 27975 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
W20250905 20:00:00.857102 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:00.818259 28715 connection.cc:232] RPC call timeout handler was delayed by -1.0778263810001363s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1841s for a response
I20250905 20:00:00.810642 29941 leader_election.cc:290] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:00.889317 30459 consensus_queue.cc:1035] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.114s
I20250905 20:00:00.781205 30496 raft_consensus.cc:3058] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:00.856070 30508 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:01.003897 30508 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:01.008096 30508 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:00.838550 28750 raft_consensus.cc:1238] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 7. Current term is 8. Ops: []
I20250905 20:00:00.839481 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:00.788652 30442 raft_consensus.cc:738] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
W20250905 20:00:00.866521 30506 log.cc:927] Time spent T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.056s user 0.001s sys 0.001s
I20250905 20:00:00.991699 27975 leader_election.cc:304] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:00.891489 30374 consensus_queue.cc:1046] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.109s
I20250905 20:00:00.866669 30381 consensus_queue.cc:1046] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.133s
I20250905 20:00:00.992070 30470 raft_consensus.cc:2747] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250905 20:00:00.888146 30495 leader_election.cc:290] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:00.817696 30488 leader_election.cc:290] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:00.907435 30420 consensus_queue.cc:1035] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.093s
I20250905 20:00:00.893747 30419 consensus_queue.cc:1046] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.145s
I20250905 20:00:00.890313 28006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1404) took 2882 ms. Trace:
I20250905 20:00:01.067361 28006 rpcz_store.cc:276] 0905 19:59:58.008029 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:59.922161 (+1914132us) service_pool.cc:225] Handling call
0905 19:59:59.951188 (+ 29027us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:00.097703 (+146515us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:00.887903 (+790200us) spinlock_profiling.cc:243] Waited 16.7 ms on lock 0x7b54000409a0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dba8550 00007fe49dc1c9b2 00007fe49dc18d96 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32
0905 20:00:00.890015 (+ 2112us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b54000104d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:00.890236 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":344439424}
W20250905 20:00:01.002782 28715 leader_election.cc:336] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.774s (SENT)
I20250905 20:00:01.010957 28008 raft_consensus.cc:2391] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 4: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
W20250905 20:00:00.980558 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:01.069205 28715 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
W20250905 20:00:01.017576 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 268816us
I20250905 20:00:01.046625 30374 raft_consensus.cc:3053] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 LEADER]: Stepping down as leader of term 7
I20250905 20:00:01.046734 30383 raft_consensus.cc:2802] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader election won for term 7
I20250905 20:00:01.051787 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:01.052938 28083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 251) took 3360 ms. Trace:
I20250905 20:00:01.053730 30451 raft_consensus.cc:3053] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 5 LEADER]: Stepping down as leader of term 5
I20250905 20:00:01.059512 27975 leader_election.cc:304] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:01.060405 30148 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Advancing to term 9
I20250905 20:00:01.061553 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:01.076102 28085 raft_consensus.cc:2408] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250905 20:00:01.062891 30471 consensus_queue.cc:1035] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 20:00:01.063155 28082 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1444) took 4096 ms. Trace:
I20250905 20:00:01.060581 30017 consensus_queue.cc:1046] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.018s
I20250905 20:00:01.077822 28082 rpcz_store.cc:276] 0905 19:59:56.967121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:57.666515 (+699394us) service_pool.cc:225] Handling call
0905 19:59:57.717340 (+ 50825us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:58.012989 (+295649us) raft_consensus.cc:1532] Early marking committed up to index 2
0905 19:59:58.492830 (+479841us) log.cc:844] Serialized 14 byte log entry
0905 20:00:00.123168 (+1630338us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:00.150920 (+ 27752us) raft_consensus.cc:1643] Marking committed up to 2
0905 20:00:00.158187 (+ 7267us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:01.062728 (+904541us) spinlock_profiling.cc:243] Waited 295 ms on lock 0x7b540007eed4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:01.062812 (+ 84us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:01.063043 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":486396,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":20927,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":14911,"spinlock_wait_cycles":679227264,"thread_start_us":1585197,"threads_started":1,"wal-append.queue_time_us":1665492}
W20250905 20:00:01.079646 28051 leader_election.cc:336] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.946s (SENT)
I20250905 20:00:01.080169 28077 raft_consensus.cc:2466] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
W20250905 20:00:01.081315 28051 leader_election.cc:336] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.954s (SENT)
I20250905 20:00:01.081931 28051 leader_election.cc:304] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:01.091984 27975 leader_election.cc:304] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:01.094647 30381 raft_consensus.cc:2747] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250905 20:00:01.097340 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:01.098112 28078 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:01.099054 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:01.100098 28077 raft_consensus.cc:2466] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 20:00:01.101518 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:01.103228 30371 raft_consensus.cc:2747] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:01.103513 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:01.104437 28085 raft_consensus.cc:2466] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
W20250905 20:00:01.108166 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:01.108397 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:01.109602 28084 raft_consensus.cc:2408] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 6 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250905 20:00:01.109979 28155 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:36164
I20250905 20:00:01.110620 28155 raft_consensus.cc:604] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Received request to transfer leadership to TS cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:00.712833 28001 rpcz_store.cc:276] 0905 19:59:57.957139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:59.853058 (+1895919us) service_pool.cc:225] Handling call
0905 19:59:59.920411 (+ 67353us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 19:59:59.936300 (+ 15889us) raft_consensus.cc:1532] Early marking committed up to index 3
0905 20:00:00.462901 (+526601us) log.cc:844] Serialized 14 byte log entry
0905 20:00:00.709562 (+246661us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:00.709619 (+ 57us) raft_consensus.cc:1643] Marking committed up to 3
0905 20:00:00.709682 (+ 63us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:00.709749 (+ 67us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:00.709868 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":592416,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":2894,"thread_start_us":682274,"threads_started":2}
I20250905 20:00:01.113219 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:01.116544 28008 raft_consensus.cc:1273] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250905 20:00:01.122963 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:01.123073 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:00.991715 30393 consensus_queue.cc:1046] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.091s
I20250905 20:00:01.138240 30496 raft_consensus.cc:2747] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250905 20:00:01.070271 30374 raft_consensus.cc:738] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
W20250905 20:00:01.145273 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 53323.7us
I20250905 20:00:01.146937 27975 leader_election.cc:304] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:01.075193 30451 raft_consensus.cc:738] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:01.160009 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:01.070845 28083 rpcz_store.cc:276] 0905 19:59:57.691992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:58.927276 (+1235284us) service_pool.cc:225] Handling call
0905 19:59:58.927816 (+ 540us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:58.928585 (+ 769us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:58.928621 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:58.929355 (+ 734us) log.cc:844] Serialized 26 byte log entry
0905 20:00:00.045095 (+1115740us) raft_consensus.cc:1643] Marking committed up to 5
0905 20:00:00.075183 (+ 30088us) log.cc:844] Serialized 14 byte log entry
0905 20:00:00.215931 (+140748us) spinlock_profiling.cc:243] Waited 5.74 ms on lock 0x7b1800150b20. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49aca450f 00007fe49aca67b1 00007fe49aca6769 00007fe49aca66f9 00007fe49aca66c1 00007fe49aca59bd 0000000000379254 0000000000379099 00007fe49ac8776a
0905 20:00:01.052286 (+836355us) spinlock_profiling.cc:243] Waited 63.7 ms on lock 0x7b5400120320. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dbabe63 00007fe49dc19d7f 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc
0905 20:00:01.052379 (+ 93us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:01.052502 (+ 123us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:01.052564 (+ 62us) raft_consensus.cc:1683] finished
0905 20:00:01.052597 (+ 33us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:01.052832 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9774,"prepare.queue_time_us":29036,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":22658,"raft.run_cpu_time_us":398,"raft.run_wall_time_us":30135,"spinlock_wait_cycles":159748224,"thread_start_us":1562904,"threads_started":2,"wal-append.queue_time_us":1179446}
I20250905 20:00:01.155812 30319 raft_consensus.cc:2802] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 20:00:01.166227 30319 raft_consensus.cc:491] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:01.164283 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20250905 20:00:01.167368 30459 raft_consensus.cc:3053] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 LEADER]: Stepping down as leader of term 5
I20250905 20:00:01.168020 28745 raft_consensus.cc:1273] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250905 20:00:01.170323 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 20:00:01.164647 28003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 2015) took 2395 ms. Trace:
I20250905 20:00:01.172501 30379 raft_consensus.cc:491] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:01.173269 30379 raft_consensus.cc:513] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:01.173828 30176 raft_consensus.cc:3053] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Stepping down as leader of term 4
W20250905 20:00:01.164944 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:01.174294 30176 raft_consensus.cc:738] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:01.175763 30379 leader_election.cc:290] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:01.175704 30176 consensus_queue.cc:260] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:01.167893 30459 raft_consensus.cc:738] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
W20250905 20:00:01.179246 30489 log.cc:927] Time spent T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.112s user 0.000s sys 0.000s
I20250905 20:00:01.164182 30471 raft_consensus.cc:491] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:01.172358 28003 rpcz_store.cc:276] 0905 19:59:58.769070 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:59.930098 (+1161028us) service_pool.cc:225] Handling call
0905 19:59:59.948855 (+ 18757us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:59.976791 (+ 27936us) raft_consensus.cc:1532] Early marking committed up to index 2
0905 20:00:00.495783 (+518992us) log.cc:844] Serialized 14 byte log entry
0905 20:00:00.707694 (+211911us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:00.721213 (+ 13519us) log.cc:844] Serialized 26 byte log entry
0905 20:00:00.721380 (+ 167us) raft_consensus.cc:1643] Marking committed up to 3
0905 20:00:00.721632 (+ 252us) log.cc:844] Serialized 14 byte log entry
0905 20:00:00.722146 (+ 514us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:00.722227 (+ 81us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:00.918786 (+196559us) spinlock_profiling.cc:243] Waited 8.61 ms on lock 0x7b180006f3a0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49aca43e2 00007fe49aca67b1 00007fe49aca6769 00007fe49aca66f9 00007fe49aca66c1 00007fe49aca59bd 0000000000379254 0000000000379099 00007fe49ac8776a
0905 20:00:01.164335 (+245549us) raft_consensus.cc:1683] finished
0905 20:00:01.164369 (+ 34us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:01.164561 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14779,"prepare.queue_time_us":602505,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":90,"raft.queue_time_us":5116,"raft.run_cpu_time_us":361,"raft.run_wall_time_us":9301,"spinlock_wait_cycles":19802880,"thread_start_us":674478,"threads_started":2,"wal-append.queue_time_us":252193}
I20250905 20:00:01.180084 30471 raft_consensus.cc:513] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:01.045308 30442 consensus_queue.cc:260] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:01.165827 30374 consensus_queue.cc:260] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:01.181185 28151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39586 (request call id 277) took 2124 ms. Trace:
I20250905 20:00:01.182211 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:01.182859 30471 leader_election.cc:290] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:01.129473 30383 raft_consensus.cc:695] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:01.182449 28151 rpcz_store.cc:276] 0905 19:59:59.056667 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:59.330540 (+273873us) service_pool.cc:225] Handling call
0905 19:59:59.331068 (+ 528us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 19:59:59.441158 (+110090us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:59:59.441209 (+ 51us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 19:59:59.813260 (+372051us) log.cc:844] Serialized 26 byte log entry
0905 20:00:00.497107 (+683847us) raft_consensus.cc:1643] Marking committed up to 6
0905 20:00:00.520210 (+ 23103us) log.cc:844] Serialized 14 byte log entry
0905 20:00:00.528390 (+ 8180us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:00.528466 (+ 76us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:01.066643 (+538177us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b180034ee80. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49aca43e2 00007fe49aca67b1 00007fe49aca6769 00007fe49aca66f9 00007fe49aca66c1 00007fe49aca59bd 0000000000379254 0000000000379099 00007fe49ac8776a
0905 20:00:01.155434 (+ 88791us) spinlock_profiling.cc:243] Waited 2.02 ms on lock 0x7b180034ee80. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49ac723d6 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49aca450f 00007fe49aca67b1
0905 20:00:01.165190 (+ 9756us) spinlock_profiling.cc:243] Waited 1.57 ms on lock 0x7b180034ee80. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49aca450f 00007fe49aca67b1 00007fe49aca6769 00007fe49aca66f9 00007fe49aca66c1 00007fe49aca59bd 0000000000379254 0000000000379099 00007fe49ac8776a
0905 20:00:01.180902 (+ 15712us) raft_consensus.cc:1683] finished
0905 20:00:01.180935 (+ 33us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:01.181102 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8808,"prepare.queue_time_us":24761,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":48,"raft.queue_time_us":24173,"raft.run_cpu_time_us":392,"raft.run_wall_time_us":18266,"spinlock_wait_cycles":33680128,"thread_start_us":578563,"threads_started":1,"wal-append.queue_time_us":689799}
I20250905 20:00:01.184145 30451 consensus_queue.cc:260] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:01.184360 30383 consensus_queue.cc:237] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:01.182283 30374 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:01.185132 30442 raft_consensus.cc:3058] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Advancing to term 3
I20250905 20:00:01.185587 30451 raft_consensus.cc:3058] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:01.166742 30319 raft_consensus.cc:3058] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
W20250905 20:00:01.188156 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:01.189042 30459 consensus_queue.cc:260] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:01.201146 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 198389us
I20250905 20:00:01.201679 30459 raft_consensus.cc:3058] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
W20250905 20:00:01.203033 28715 leader_election.cc:336] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 2.114s (SENT)
W20250905 20:00:01.194581 30484 log.cc:927] Time spent T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.402s user 0.003s sys 0.002s
I20250905 20:00:01.200346 30508 consensus_queue.cc:1035] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250905 20:00:01.191603 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:01.204465 28153 raft_consensus.cc:2466] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
I20250905 20:00:01.171878 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 20:00:01.207863 30017 raft_consensus.cc:2802] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 20:00:01.208720 30017 raft_consensus.cc:491] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:01.209224 30017 raft_consensus.cc:3058] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:01.158353 30419 raft_consensus.cc:3053] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Stepping down as leader of term 3
W20250905 20:00:01.210036 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 44013.9us
I20250905 20:00:01.210304 30419 raft_consensus.cc:738] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:01.211292 30419 consensus_queue.cc:260] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:01.212509 30371 consensus_queue.cc:1046] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20250905 20:00:01.213555 30515 log.cc:927] Time spent T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.120s user 0.000s sys 0.001s
I20250905 20:00:01.217195 30371 raft_consensus.cc:3053] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 LEADER]: Stepping down as leader of term 7
I20250905 20:00:01.217722 30371 raft_consensus.cc:738] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:01.218284 30176 raft_consensus.cc:3058] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:01.218699 30371 consensus_queue.cc:260] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:01.220170 30371 raft_consensus.cc:3058] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:01.220803 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:01.214213 30470 consensus_queue.cc:1035] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250905 20:00:01.222882 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:01.223955 28745 raft_consensus.cc:2408] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20250905 20:00:01.225358 30240 consensus_meta.cc:287] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.551s user 0.011s sys 0.000s
W20250905 20:00:01.208025 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 61319.4us
I20250905 20:00:01.213109 30319 raft_consensus.cc:513] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:01.227669 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 146398us
I20250905 20:00:01.203545 28715 leader_election.cc:304] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:01.228864 27975 leader_election.cc:304] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
W20250905 20:00:01.245425 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:01.239086 30372 raft_consensus.cc:2747] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:01.247838 30196 raft_consensus.cc:491] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:01.251303 30196 raft_consensus.cc:513] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:01.249400 27975 leader_election.cc:304] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:01.253749 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:01.255398 27973 leader_election.cc:304] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:01.254545 28155 raft_consensus.cc:2466] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
I20250905 20:00:01.265336 30317 raft_consensus.cc:2802] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:01.265460 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:01.266242 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:01.247663 30346 raft_consensus.cc:2747] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250905 20:00:01.274571 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 71579.5us
I20250905 20:00:01.275573 28152 raft_consensus.cc:1273] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250905 20:00:01.271098 28050 leader_election.cc:304] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:01.269959 28151 raft_consensus.cc:2466] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
I20250905 20:00:01.274816 27973 leader_election.cc:304] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:01.290688 28155 raft_consensus.cc:1273] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250905 20:00:01.291568 30017 raft_consensus.cc:513] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:01.294345 28150 raft_consensus.cc:1273] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20250905 20:00:01.303438 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 410106us
I20250905 20:00:01.263734 30372 raft_consensus.cc:2762] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20250905 20:00:01.312811 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:01.314067 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:01.342115 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:01.269052 30317 raft_consensus.cc:491] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:01.341817 28050 leader_election.cc:304] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:01.353108 30196 leader_election.cc:290] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:01.346406 30488 raft_consensus.cc:2802] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 20:00:01.353824 30488 raft_consensus.cc:491] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:01.354187 30488 raft_consensus.cc:3058] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:01.355976 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:01.356848 28083 raft_consensus.cc:2391] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 3: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:01.350001 30317 raft_consensus.cc:3058] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:01.361166 27907 catalog_manager.cc:5582] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 2 to 3, leader changed from 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 3 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } }
I20250905 20:00:01.362669 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:01.306174 30017 leader_election.cc:290] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:01.373893 30196 raft_consensus.cc:2802] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 20:00:01.387058 28083 raft_consensus.cc:1238] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250905 20:00:01.372566 30471 raft_consensus.cc:2802] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election won for term 6
I20250905 20:00:01.403409 28050 leader_election.cc:304] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:01.395421 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:01.388741 27907 catalog_manager.cc:5582] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 2 to 4, leader changed from 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) to 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132). New cstate: current_term: 4 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 20:00:01.376116 30420 raft_consensus.cc:491] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:01.399348 28153 raft_consensus.cc:1273] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250905 20:00:01.400139 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:01.411106 28151 raft_consensus.cc:2408] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250905 20:00:01.384800 30196 raft_consensus.cc:491] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:01.414924 28084 raft_consensus.cc:1273] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250905 20:00:01.412158 28158 raft_consensus.cc:2466] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 20:00:01.419309 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:01.425031 27908 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34440 (request call id 253) took 8388 ms. Trace:
I20250905 20:00:01.412055 30379 raft_consensus.cc:2802] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 20:00:01.424278 28079 raft_consensus.cc:2466] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 20:00:01.435040 28082 raft_consensus.cc:1238] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20250905 20:00:01.420145 30471 raft_consensus.cc:695] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:01.441138 30471 consensus_queue.cc:237] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:01.443751 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:01.425740 27908 rpcz_store.cc:276] 0905 19:59:53.029685 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:53.029961 (+ 276us) service_pool.cc:225] Handling call
0905 20:00:01.411593 (+8381632us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:59:56.023472 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:59:56.230318 (+206846us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:59:56.239942 (+ 9624us) write_op.cc:435] Acquired schema lock
0905 19:59:56.243452 (+ 3510us) tablet.cc:660] Decoding operations
0905 19:59:56.518422 (+274970us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:59:56.523508 (+ 5086us) write_op.cc:641] Partition lock acquired for write op
0905 19:59:56.526620 (+ 3112us) tablet.cc:683] Acquiring locks for 12 operations
0905 19:59:56.811092 (+284472us) tablet.cc:699] Row locks acquired
0905 19:59:56.813882 (+ 2790us) write_op.cc:260] PREPARE: finished
0905 19:59:56.824893 (+ 11011us) write_op.cc:270] Start()
0905 19:59:56.855207 (+ 30314us) write_op.cc:276] Timestamp: P: 1757102396824788 usec, L: 0
0905 19:59:56.857767 (+ 2560us) op_driver.cc:348] REPLICATION: starting
0905 19:59:57.167513 (+309746us) log.cc:844] Serialized 4785 byte log entry
0905 20:00:00.182036 (+3014523us) op_driver.cc:464] REPLICATION: finished
0905 20:00:00.464323 (+282287us) write_op.cc:301] APPLY: starting
0905 20:00:00.488267 (+ 23944us) tablet.cc:1365] starting BulkCheckPresence
0905 20:00:00.521315 (+ 33048us) tablet.cc:1368] finished BulkCheckPresence
0905 20:00:00.521466 (+ 151us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 20:00:00.582201 (+ 60735us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 20:00:00.605428 (+ 23227us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=12
0905 20:00:00.605529 (+ 101us) write_op.cc:312] APPLY: finished
0905 20:00:00.606238 (+ 709us) log.cc:844] Serialized 89 byte log entry
0905 20:00:00.625155 (+ 18917us) write_op.cc:489] Releasing partition, row and schema locks
0905 20:00:00.706038 (+ 80883us) write_op.cc:454] Released schema lock
0905 20:00:00.720824 (+ 14786us) write_op.cc:341] FINISH: Updating metrics
0905 20:00:00.855600 (+134776us) write_op.cc:489] Releasing partition, row and schema locks
0905 20:00:00.865815 (+ 10215us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1882,"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":161444,"apply.run_cpu_time_us":6907,"apply.run_wall_time_us":379631,"num_ops":12,"prepare.queue_time_us":931641,"prepare.run_cpu_time_us":34468,"prepare.run_wall_time_us":1793902,"raft.queue_time_us":1527690,"raft.run_cpu_time_us":41936,"raft.run_wall_time_us":85149,"replication_time_us":3324003,"thread_start_us":2199842,"threads_started":4,"wal-append.queue_time_us":511056}]]}
W20250905 20:00:01.458611 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:01.465826 30419 raft_consensus.cc:3058] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:01.420250 30196 raft_consensus.cc:3058] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:01.402587 30393 consensus_queue.cc:1046] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.071s
I20250905 20:00:01.484172 28084 raft_consensus.cc:1238] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250905 20:00:01.414603 30420 raft_consensus.cc:513] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:01.464699 28050 leader_election.cc:304] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:01.484925 27907 catalog_manager.cc:5582] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 2 to 3, leader changed from cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 3 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } }
I20250905 20:00:01.428550 30379 raft_consensus.cc:491] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:01.461025 30319 leader_election.cc:290] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:01.494534 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:01.496556 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:01.509250 28082 raft_consensus.cc:1273] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250905 20:00:01.503185 27907 catalog_manager.cc:5582] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 2 to 3, leader changed from cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 3 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20250905 20:00:01.492703 30379 raft_consensus.cc:3058] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:01.490948 30495 raft_consensus.cc:2802] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 20:00:01.520076 28158 raft_consensus.cc:1273] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250905 20:00:01.497289 30176 raft_consensus.cc:491] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:01.531476 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 20:00:01.532361 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:01.532471 28151 raft_consensus.cc:3058] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:01.535281 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:01.536082 28153 raft_consensus.cc:3058] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:01.539320 28084 raft_consensus.cc:1238] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250905 20:00:01.498270 30240 consensus_queue.cc:1035] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.036s
I20250905 20:00:01.520690 30470 raft_consensus.cc:491] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:01.544198 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:01.549386 28082 raft_consensus.cc:3058] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:01.515259 30017 raft_consensus.cc:491] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
W20250905 20:00:01.550654 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:01.547479 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:01.547953 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:01.567379 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:01.567930 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:01.538473 27975 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:01.522699 30495 raft_consensus.cc:491] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:01.563591 27907 catalog_manager.cc:5582] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 1 to 2, leader changed from e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 2 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20250905 20:00:01.604039 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:01.605000 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 20:00:01.573302 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 298559us
I20250905 20:00:01.608337 27973 leader_election.cc:304] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:01.544960 30501 consensus_queue.cc:1035] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.055s
I20250905 20:00:01.588079 30495 raft_consensus.cc:3058] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:01.546700 30459 consensus_queue.cc:1035] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.058s
I20250905 20:00:01.617137 28084 raft_consensus.cc:2391] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 4: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:01.566118 30383 consensus_queue.cc:1035] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.049s
I20250905 20:00:01.571472 30319 consensus_queue.cc:1035] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20250905 20:00:01.571569 30017 raft_consensus.cc:513] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:01.585754 30176 raft_consensus.cc:513] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:01.623868 27907 catalog_manager.cc:5582] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 1 to 4, leader changed from 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) to 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132). New cstate: current_term: 4 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 20:00:01.638581 27973 leader_election.cc:304] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:01.585086 30470 raft_consensus.cc:513] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:01.658509 28153 raft_consensus.cc:2466] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:01.626230 30529 raft_consensus.cc:2747] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:01.660926 27907 catalog_manager.cc:5582] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 2 to 3, leader changed from cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 3 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } }
I20250905 20:00:01.668511 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:01.670168 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:01.635511 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:01.684543 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:01.687649 30530 consensus_queue.cc:1035] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.014s
I20250905 20:00:01.687901 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 20:00:01.670843 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:01.692783 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 90617.9us
I20250905 20:00:01.694736 27975 leader_election.cc:304] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a
W20250905 20:00:01.684428 30520 log.cc:927] Time spent T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.136s user 0.000s sys 0.001s
I20250905 20:00:01.679998 27907 catalog_manager.cc:5582] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 20:00:01.677495 30420 leader_election.cc:290] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:01.690042 30372 raft_consensus.cc:2747] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 20:00:01.707031 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:01.690901 30393 raft_consensus.cc:491] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:01.699143 30529 raft_consensus.cc:2747] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 20:00:01.703464 30402 raft_consensus.cc:2802] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election won for term 5
W20250905 20:00:01.705951 30519 log.cc:927] Time spent T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.186s user 0.001s sys 0.000s
I20250905 20:00:01.721343 27907 catalog_manager.cc:5582] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20250905 20:00:01.719447 30451 raft_consensus.cc:491] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:01.732990 28151 raft_consensus.cc:2466] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:01.735466 27975 leader_election.cc:304] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a
W20250905 20:00:01.740372 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:01.740715 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:01.739750 27907 catalog_manager.cc:5582] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 2 to 5, leader changed from 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) to 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131). New cstate: current_term: 5 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 20:00:01.703775 30263 consensus_queue.cc:1035] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.020s
I20250905 20:00:01.753746 28151 raft_consensus.cc:1273] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20250905 20:00:01.757203 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 261572us
W20250905 20:00:01.770102 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 103266us
I20250905 20:00:01.771843 28085 raft_consensus.cc:1273] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250905 20:00:01.714340 30393 raft_consensus.cc:513] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:01.739181 30402 raft_consensus.cc:695] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:01.786015 27907 catalog_manager.cc:5582] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
W20250905 20:00:01.773944 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:01.747429 30232 raft_consensus.cc:2802] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election won for term 5
I20250905 20:00:01.793138 27907 catalog_manager.cc:5582] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 1 to 3, leader changed from 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) to 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132). New cstate: current_term: 3 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
W20250905 20:00:01.769496 30148 consensus_meta.cc:287] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.636s user 0.012s sys 0.001s
I20250905 20:00:01.767992 30517 raft_consensus.cc:3053] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Stepping down as leader of term 3
W20250905 20:00:01.808658 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 52396.7us
I20250905 20:00:01.738122 30451 raft_consensus.cc:513] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:01.837356 28009 raft_consensus.cc:2408] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20250905 20:00:01.837985 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 211395us
W20250905 20:00:01.840689 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 72714us
I20250905 20:00:01.841063 27907 catalog_manager.cc:5582] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20250905 20:00:01.800413 30176 leader_election.cc:290] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:01.849198 28125 leader_election.cc:304] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:01.851507 28085 raft_consensus.cc:1273] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250905 20:00:01.854176 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:01.815524 30017 leader_election.cc:290] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:01.837213 30523 log.cc:927] Time spent T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.213s user 0.001s sys 0.000s
I20250905 20:00:01.862020 27907 catalog_manager.cc:5582] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 1 to 3, leader changed from cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 3 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } }
I20250905 20:00:01.876981 27907 catalog_manager.cc:5582] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20250905 20:00:01.815392 30517 raft_consensus.cc:738] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:01.887248 28125 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:01.888558 28125 leader_election.cc:304] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:01.888105 27907 catalog_manager.cc:5582] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } }
I20250905 20:00:01.816560 30501 consensus_queue.cc:1035] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20250905 20:00:01.898239 28746 raft_consensus.cc:1273] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20250905 20:00:01.905193 30535 log.cc:927] Time spent T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20250905 20:00:01.904385 30517 consensus_queue.cc:260] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:01.905995 30263 raft_consensus.cc:491] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:01.905680 30402 consensus_queue.cc:237] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:01.907974 30522 log.cc:927] Time spent T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.224s user 0.002s sys 0.000s
W20250905 20:00:01.913257 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 74115.1us
W20250905 20:00:01.915225 30536 log.cc:927] Time spent T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.099s user 0.001s sys 0.000s
W20250905 20:00:01.915863 30533 log.cc:927] Time spent T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
W20250905 20:00:01.916030 30531 log.cc:927] Time spent T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.167s user 0.000s sys 0.000s
W20250905 20:00:01.917519 30534 log.cc:927] Time spent T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.164s user 0.002s sys 0.000s
I20250905 20:00:01.922088 30393 leader_election.cc:290] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 20:00:01.923273 30532 log.cc:927] Time spent T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.219s user 0.001s sys 0.000s
I20250905 20:00:01.892879 30232 raft_consensus.cc:695] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:01.897990 30374 raft_consensus.cc:2747] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250905 20:00:01.932814 30317 consensus_meta.cc:287] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.574s user 0.010s sys 0.002s
I20250905 20:00:02.013795 30317 raft_consensus.cc:513] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:01.935284 30552 raft_consensus.cc:2747] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:02.016251 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:01.942641 30451 leader_election.cc:290] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 20:00:01.951463 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 62995.7us
W20250905 20:00:01.891357 30525 log.cc:927] Time spent T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.229s user 0.000s sys 0.001s
W20250905 20:00:01.983728 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:01.944319 30420 raft_consensus.cc:2747] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250905 20:00:01.989092 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:02.019726 28125 leader_election.cc:304] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:02.000262 28084 raft_consensus.cc:1238] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250905 20:00:02.020473 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:01.890920 30470 leader_election.cc:290] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:02.021037 28125 leader_election.cc:304] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 20:00:02.006536 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:02.021494 30420 raft_consensus.cc:2802] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election won for term 4
W20250905 20:00:02.021806 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:01.937135 28009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 288) took 1629 ms. Trace:
I20250905 20:00:02.001394 27973 leader_election.cc:304] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:01.845305 30538 consensus_queue.cc:1035] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.151s
W20250905 20:00:02.022234 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:02.022660 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:02.022994 30374 raft_consensus.cc:2802] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 20:00:02.022106 30420 raft_consensus.cc:695] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:02.023690 30374 raft_consensus.cc:695] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:02.022236 28009 rpcz_store.cc:276] 0905 20:00:00.307902 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:00.309301 (+ 1399us) service_pool.cc:225] Handling call
0905 20:00:01.936846 (+1627545us) spinlock_profiling.cc:243] Waited 1.53 s on lock 0x7b54000546d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:01.937029 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3510818560}
I20250905 20:00:02.022065 28125 leader_election.cc:304] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:02.031096 30317 leader_election.cc:290] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:02.033051 27973 leader_election.cc:304] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:02.034911 30544 raft_consensus.cc:2747] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:02.026861 30529 consensus_queue.cc:1035] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.009s
I20250905 20:00:02.034515 30232 consensus_queue.cc:237] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:02.036021 30420 consensus_queue.cc:237] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:02.046963 27973 leader_election.cc:304] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
W20250905 20:00:02.049881 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:02.055562 28125 leader_election.cc:304] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:02.089416 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
W20250905 20:00:02.039824 30419 consensus_meta.cc:287] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.573s user 0.014s sys 0.000s
I20250905 20:00:02.092072 30557 raft_consensus.cc:2747] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:02.104946 30551 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:02.105705 30379 consensus_meta.cc:287] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.573s user 0.010s sys 0.000s
I20250905 20:00:02.088589 30374 consensus_queue.cc:237] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:02.109735 30530 raft_consensus.cc:2802] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250905 20:00:02.113031 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:02.113489 30196 consensus_meta.cc:287] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.630s user 0.008s sys 0.005s
I20250905 20:00:02.090412 30459 raft_consensus.cc:2747] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250905 20:00:02.114751 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 67904.5us
I20250905 20:00:02.100167 30470 raft_consensus.cc:2747] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:02.120854 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 20:00:02.124186 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:02.028818 30545 raft_consensus.cc:491] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:02.124635 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 69217.7us
W20250905 20:00:02.128160 28082 consensus_meta.cc:287] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.578s user 0.013s sys 0.000s
I20250905 20:00:02.045408 30517 raft_consensus.cc:3058] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:02.011255 30263 raft_consensus.cc:513] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:02.138164 28082 raft_consensus.cc:2408] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20250905 20:00:02.141394 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:02.140445 30196 raft_consensus.cc:513] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:02.149709 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:02.138512 30530 raft_consensus.cc:491] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:02.150799 30530 raft_consensus.cc:3058] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:02.151221 28084 raft_consensus.cc:1273] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250905 20:00:02.151383 30196 leader_election.cc:290] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:02.152401 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:02.152583 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:02.154563 28080 raft_consensus.cc:2466] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 20:00:02.155781 28085 raft_consensus.cc:2366] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5.
I20250905 20:00:02.153770 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:02.135969 30551 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:02.161964 30263 leader_election.cc:290] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:02.164326 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:02.148847 30459 raft_consensus.cc:491] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:02.134827 30488 consensus_meta.cc:287] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.780s user 0.005s sys 0.008s
W20250905 20:00:02.179759 28122 leader_election.cc:336] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.753s (SENT)
I20250905 20:00:02.180456 28122 leader_election.cc:304] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:02.169584 28084 raft_consensus.cc:2466] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
I20250905 20:00:02.183470 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:02.184024 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:02.184231 28750 raft_consensus.cc:2466] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 7.
I20250905 20:00:02.185350 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:02.162077 30551 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:02.186782 27973 leader_election.cc:304] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:02.186269 28080 raft_consensus.cc:2391] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 3: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
W20250905 20:00:02.188071 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:02.187947 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:02.188421 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:02.189132 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:02.189874 28049 leader_election.cc:304] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:02.193836 28153 raft_consensus.cc:3058] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:02.194095 28082 raft_consensus.cc:2466] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 20:00:02.194622 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:02.195118 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:02.195152 28150 raft_consensus.cc:2466] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 7.
I20250905 20:00:02.197595 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:02.200048 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:02.192325 30557 raft_consensus.cc:2747] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250905 20:00:02.135504 30379 raft_consensus.cc:513] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:02.204689 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:02.207792 27973 leader_election.cc:304] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:02.208197 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:02.166968 30545 raft_consensus.cc:513] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:02.209343 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:02.209499 30539 raft_consensus.cc:3058] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:02.209946 28080 raft_consensus.cc:2374] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4. Re-sending same reply.
I20250905 20:00:02.210999 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:02.211668 28080 raft_consensus.cc:2466] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 20:00:02.211668 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:02.213050 28079 raft_consensus.cc:2408] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250905 20:00:02.208871 28082 raft_consensus.cc:2466] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
I20250905 20:00:02.211112 30545 leader_election.cc:290] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:02.226529 27973 leader_election.cc:304] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:02.232066 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:02.290488 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:02.201718 30551 raft_consensus.cc:2802] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250905 20:00:02.300678 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 73897.4us
I20250905 20:00:02.301599 27973 leader_election.cc:304] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
W20250905 20:00:02.302313 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 71415.2us
I20250905 20:00:02.308274 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:02.308904 28122 leader_election.cc:336] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.572s (SENT)
I20250905 20:00:02.191221 30488 raft_consensus.cc:513] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:02.309762 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:02.190404 30459 raft_consensus.cc:513] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:02.309886 27973 leader_election.cc:304] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:02.311853 28081 raft_consensus.cc:1238] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:02.312908 27894 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet e707b0f561934674947fb463ce626a07 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 10)
I20250905 20:00:02.234637 30017 raft_consensus.cc:2802] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 20:00:02.330690 30470 raft_consensus.cc:2802] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:02.331228 30470 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:02.299944 30543 raft_consensus.cc:491] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:02.338747 30419 consensus_queue.cc:1046] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20250905 20:00:02.339357 30543 raft_consensus.cc:513] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:02.339864 30419 consensus_queue.cc:1035] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 20:00:02.310724 30545 raft_consensus.cc:2802] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250905 20:00:02.342474 28122 leader_election.cc:336] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.627s (SENT)
I20250905 20:00:02.343050 28122 leader_election.cc:304] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:02.342873 28151 raft_consensus.cc:1273] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20250905 20:00:02.345988 28151 pending_rounds.cc:85] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting all ops after (but not including) 2
I20250905 20:00:02.346323 28151 pending_rounds.cc:107] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20250905 20:00:02.346664 28151 raft_consensus.cc:2887] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250905 20:00:02.313210 30317 raft_consensus.cc:2747] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:02.337992 30488 leader_election.cc:290] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:02.331681 30470 raft_consensus.cc:3058] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:02.341651 30419 raft_consensus.cc:3053] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Stepping down as leader of term 4
I20250905 20:00:02.306514 30551 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:02.355425 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:02.365686 28153 raft_consensus.cc:2466] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
I20250905 20:00:02.355345 30419 raft_consensus.cc:738] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:02.355230 30557 raft_consensus.cc:2747] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:02.367879 30419 consensus_queue.cc:260] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:02.372973 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:02.373955 28008 raft_consensus.cc:2466] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
W20250905 20:00:02.376482 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 59684.6us
I20250905 20:00:02.377872 27973 leader_election.cc:304] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:02.379278 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:02.379614 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:02.382134 28156 raft_consensus.cc:1273] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20250905 20:00:02.359540 30551 raft_consensus.cc:3058] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:02.382730 28156 pending_rounds.cc:85] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting all ops after (but not including) 4
I20250905 20:00:02.383598 28156 pending_rounds.cc:107] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20250905 20:00:02.383525 28077 raft_consensus.cc:2366] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5.
I20250905 20:00:02.383729 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:02.384128 28156 raft_consensus.cc:2887] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250905 20:00:02.385491 30561 log.cc:927] Time spent T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20250905 20:00:02.341804 30545 raft_consensus.cc:491] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:02.347208 30017 raft_consensus.cc:491] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:02.393675 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:02.394749 30557 consensus_queue.cc:1035] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20250905 20:00:02.395493 30319 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:02.396131 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:02.397519 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:02.424808 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:02.397742 30459 leader_election.cc:290] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:02.399045 30442 consensus_queue.cc:1035] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250905 20:00:02.406775 30374 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:02.412865 30495 consensus_meta.cc:287] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.775s user 0.012s sys 0.000s
I20250905 20:00:02.396149 30176 raft_consensus.cc:491] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:02.421633 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:02.461791 28156 raft_consensus.cc:3058] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
W20250905 20:00:02.382622 30565 log.cc:927] Time spent T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
W20250905 20:00:02.426357 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 48573.4us
I20250905 20:00:02.426941 28150 raft_consensus.cc:1238] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:02.418524 28001 raft_consensus.cc:2374] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5. Re-sending same reply.
I20250905 20:00:02.429159 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:02.431432 28159 raft_consensus.cc:1238] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:02.433898 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:02.435458 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:02.376155 30263 raft_consensus.cc:491] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
W20250905 20:00:02.437111 28122 leader_election.cc:336] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.710s (SENT)
W20250905 20:00:02.473291 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:02.473119 27973 leader_election.cc:304] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:02.432582 28050 leader_election.cc:304] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:02.438375 28157 raft_consensus.cc:1238] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 2. Current term is 3. Ops: []
I20250905 20:00:02.406286 30539 raft_consensus.cc:2747] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 20:00:02.439723 30374 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:02.430974 30319 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:02.423509 30017 raft_consensus.cc:3058] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
W20250905 20:00:02.436378 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:02.446755 30495 raft_consensus.cc:513] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:02.448277 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:02.486744 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:02.485738 28156 raft_consensus.cc:2466] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
I20250905 20:00:02.487536 28001 raft_consensus.cc:2466] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
W20250905 20:00:02.488121 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 107879us
I20250905 20:00:02.451151 30470 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:02.460218 30379 leader_election.cc:290] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:02.414376 30545 raft_consensus.cc:3058] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:02.470283 28008 raft_consensus.cc:2408] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20250905 20:00:02.476297 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 43777.5us
I20250905 20:00:02.478562 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:02.481338 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:02.468283 30402 raft_consensus.cc:2747] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:02.432366 30543 leader_election.cc:290] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:02.495930 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:02.480334 30196 raft_consensus.cc:2802] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election won for term 5
I20250905 20:00:02.486395 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:02.450657 30551 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:02.493566 28156 raft_consensus.cc:1215] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Deduplicated request from leader. Original: 2.5->[5.6-5.6] Dedup: 5.6->[]
I20250905 20:00:02.495865 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:02.500016 28156 raft_consensus.cc:1238] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20250905 20:00:02.483798 30471 raft_consensus.cc:2802] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 20:00:02.502756 28005 raft_consensus.cc:2466] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
I20250905 20:00:02.502326 28083 raft_consensus.cc:2391] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 4: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:02.503813 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:02.503787 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:02.506086 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:02.506897 28004 raft_consensus.cc:2466] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
I20250905 20:00:02.482661 30501 consensus_queue.cc:1035] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.016s
I20250905 20:00:02.477761 30176 raft_consensus.cc:513] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:02.510069 28078 raft_consensus.cc:2466] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 20:00:02.511080 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:02.511169 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:02.511744 28078 raft_consensus.cc:2466] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 20:00:02.511860 28080 raft_consensus.cc:2466] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 20:00:02.512605 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:02.512843 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:02.513218 28078 raft_consensus.cc:3058] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:02.513790 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:02.514919 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:02.515534 28006 raft_consensus.cc:2466] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
I20250905 20:00:02.518308 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:02.518060 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:02.520092 28079 raft_consensus.cc:1273] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250905 20:00:02.526854 28079 raft_consensus.cc:1273] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250905 20:00:02.528236 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:02.518515 28050 leader_election.cc:304] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:02.524099 28002 raft_consensus.cc:2374] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4. Re-sending same reply.
I20250905 20:00:02.532814 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:02.523562 28010 raft_consensus.cc:2466] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 20:00:02.535760 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:02.505074 30471 raft_consensus.cc:491] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:02.536679 28008 raft_consensus.cc:2466] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 8.
I20250905 20:00:02.540069 30176 leader_election.cc:290] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:02.529219 30317 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:02.542307 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:02.542862 28158 raft_consensus.cc:2466] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 20:00:02.543458 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:02.544777 30374 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:02.544956 30551 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:02.537580 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:02.556842 28004 raft_consensus.cc:2408] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250905 20:00:02.488426 30263 raft_consensus.cc:513] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:02.551085 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:02.563686 28006 raft_consensus.cc:2466] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 20:00:02.552122 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:02.564713 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:02.507675 30196 raft_consensus.cc:695] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 5 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:02.558869 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:02.559922 28084 raft_consensus.cc:2408] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250905 20:00:02.559624 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:02.565299 28153 raft_consensus.cc:3058] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Advancing to term 8
W20250905 20:00:02.575758 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:02.565724 28080 raft_consensus.cc:2466] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:02.581164 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:02.581977 28006 raft_consensus.cc:2466] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 20:00:02.582536 28078 raft_consensus.cc:2466] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
I20250905 20:00:02.546746 30379 raft_consensus.cc:2802] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election won for term 5
I20250905 20:00:02.563012 30317 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:02.586616 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:02.587208 30319 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:02.587855 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:02.589092 30196 consensus_queue.cc:237] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:02.550379 30471 raft_consensus.cc:3058] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:02.600265 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:02.603904 28010 raft_consensus.cc:3058] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:02.598650 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:02.602084 28001 raft_consensus.cc:2391] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 5: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 20:00:02.590296 30379 raft_consensus.cc:695] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:02.606813 28156 raft_consensus.cc:2466] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 20:00:02.608577 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:02.609608 28157 raft_consensus.cc:2466] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 20:00:02.612421 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:02.613274 28007 raft_consensus.cc:2366] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 6. Current term is 7.
I20250905 20:00:02.613533 30317 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:02.614858 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:02.615495 28151 raft_consensus.cc:2466] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:02.615542 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:02.614178 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:02.621574 28081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1493) took 1242 ms. Trace:
I20250905 20:00:02.621918 28081 rpcz_store.cc:276] 0905 20:00:01.378843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:01.389357 (+ 10514us) service_pool.cc:225] Handling call
0905 20:00:01.412476 (+ 23119us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:02.316272 (+903796us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:02.621396 (+305124us) spinlock_profiling.cc:243] Waited 893 ms on lock 0x7b540023c154. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:02.621529 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2053651840}
I20250905 20:00:02.624429 27973 leader_election.cc:304] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 20:00:02.611862 30573 log.cc:927] Time spent T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
I20250905 20:00:02.550375 30420 consensus_queue.cc:1035] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20250905 20:00:02.626720 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:02.626374 28078 raft_consensus.cc:1273] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250905 20:00:02.630749 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 20:00:02.634567 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:02.635862 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:02.643126 30232 raft_consensus.cc:491] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:02.644009 30232 raft_consensus.cc:513] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:02.650118 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:02.651055 28079 raft_consensus.cc:2466] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:02.648633 28084 raft_consensus.cc:2466] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 20:00:02.616861 30379 consensus_queue.cc:237] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:02.657486 28007 raft_consensus.cc:1238] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250905 20:00:02.655303 27973 leader_election.cc:304] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:02.660873 28153 raft_consensus.cc:2466] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 8.
I20250905 20:00:02.667375 30495 leader_election.cc:290] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:02.670764 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:02.671629 28747 raft_consensus.cc:3058] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:02.675973 28122 leader_election.cc:304] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:02.678120 28122 leader_election.cc:304] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:02.681197 28010 raft_consensus.cc:2466] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
I20250905 20:00:02.695030 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 20:00:02.696789 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:02.647035 30232 leader_election.cc:290] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:02.621266 30374 consensus_queue.cc:1035] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20250905 20:00:02.653366 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:02.630867 30530 raft_consensus.cc:513] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:02.725082 28122 leader_election.cc:304] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:02.655697 30402 raft_consensus.cc:2802] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 20:00:02.715889 28747 raft_consensus.cc:2466] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 8.
I20250905 20:00:02.718403 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:02.721899 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:02.732743 28049 leader_election.cc:304] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:02.734190 30495 raft_consensus.cc:2802] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader election won for term 8
I20250905 20:00:02.735507 30495 raft_consensus.cc:695] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 8 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:02.736740 30495 consensus_queue.cc:237] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:02.724272 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 173901us
I20250905 20:00:02.726608 28125 leader_election.cc:304] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:02.693326 30319 raft_consensus.cc:2802] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 20:00:02.647780 30240 consensus_queue.cc:1035] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
I20250905 20:00:02.749727 28122 leader_election.cc:304] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:02.755833 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 20:00:02.706398 30459 raft_consensus.cc:2802] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 20:00:02.653100 30529 consensus_queue.cc:1035] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.063s
W20250905 20:00:02.743918 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 88719.8us
I20250905 20:00:02.770634 27973 leader_election.cc:304] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:02.729230 30470 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:02.766395 28051 leader_election.cc:336] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 2.105s (SENT)
I20250905 20:00:02.741667 30557 raft_consensus.cc:2747] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 20:00:02.743217 30402 raft_consensus.cc:491] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:02.755472 30319 raft_consensus.cc:491] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250905 20:00:02.805480 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:02.742753 28745 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 9.
I20250905 20:00:02.764159 30459 raft_consensus.cc:491] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:02.703742 30554 consensus_queue.cc:1035] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.070s
W20250905 20:00:02.819832 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:02.806269 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:02.812232 30393 raft_consensus.cc:2747] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:02.811374 30402 raft_consensus.cc:3058] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:02.813856 30319 raft_consensus.cc:3058] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:02.813329 30420 raft_consensus.cc:2802] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election won for term 4
W20250905 20:00:02.842506 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 76173.8us
W20250905 20:00:02.832306 30567 log.cc:927] Time spent T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.326s user 0.000s sys 0.001s
W20250905 20:00:02.831584 30520 log.cc:927] Time spent T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.058s user 0.000s sys 0.001s
W20250905 20:00:02.848675 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 122221us
I20250905 20:00:02.835202 30459 raft_consensus.cc:3058] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:02.835299 30374 raft_consensus.cc:491] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
W20250905 20:00:02.857019 28051 leader_election.cc:336] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 2.149s (SENT)
I20250905 20:00:02.844992 30148 raft_consensus.cc:2802] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 20:00:02.867797 30575 consensus_queue.cc:1035] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20250905 20:00:02.869462 30593 raft_consensus.cc:491] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:02.884799 28125 leader_election.cc:304] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:02.854511 30263 leader_election.cc:290] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:02.903084 30530 leader_election.cc:290] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:02.904117 30586 raft_consensus.cc:2802] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:02.904753 30586 raft_consensus.cc:491] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:02.905184 30586 raft_consensus.cc:3058] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:02.848006 30470 consensus_queue.cc:1035] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.038s
W20250905 20:00:02.875720 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 134840us
W20250905 20:00:02.876770 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 194053us
I20250905 20:00:02.864013 30420 raft_consensus.cc:695] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:02.897833 30017 raft_consensus.cc:513] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:02.907698 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:02.981159 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 231540us
W20250905 20:00:03.036110 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:03.056540 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 197994us
W20250905 20:00:03.057040 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 200056us
W20250905 20:00:03.058020 28051 connection.cc:232] RPC call timeout handler was delayed by -4.3083898710000312s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2784s for a response
W20250905 20:00:02.978777 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 172019us
I20250905 20:00:02.879998 30148 raft_consensus.cc:491] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:03.061136 27975 leader_election.cc:304] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:03.063952 28122 leader_election.cc:304] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:03.058238 28049 leader_election.cc:304] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:03.064630 28076 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1473) took 1045 ms. Trace:
I20250905 20:00:03.065167 28076 rpcz_store.cc:276] 0905 20:00:02.012872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:02.907243 (+894371us) service_pool.cc:225] Handling call
0905 20:00:03.056253 (+149010us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:03.079746 28125 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:03.085826 28009 raft_consensus.cc:2391] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:03.083782 28717 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:03.085963 28122 leader_election.cc:304] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:03.018203 30442 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:02.880950 30374 raft_consensus.cc:513] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:02.908313 30393 raft_consensus.cc:2802] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 20:00:03.067818 30148 raft_consensus.cc:3058] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:03.067845 30196 raft_consensus.cc:2747] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250905 20:00:03.093480 28051 connection.cc:232] RPC call timeout handler was delayed by -4.2078205300000491s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22080000000000002s for a response
I20250905 20:00:03.094743 28125 leader_election.cc:304] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:03.102260 28125 leader_election.cc:304] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:02.890193 30593 raft_consensus.cc:513] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:03.095832 30594 raft_consensus.cc:2802] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 20:00:03.093158 30554 raft_consensus.cc:2802] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 20:00:03.115602 30559 raft_consensus.cc:491] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:03.116258 30537 raft_consensus.cc:2747] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250905 20:00:03.116011 30420 consensus_queue.cc:237] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:03.116837 30604 consensus_queue.cc:1046] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250905 20:00:03.093143 30393 raft_consensus.cc:491] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:03.117159 30196 raft_consensus.cc:491] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:03.093305 30442 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:03.135471 28004 raft_consensus.cc:1238] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 4. Current term is 5. Ops: []
W20250905 20:00:03.114240 30581 log.cc:927] Time spent T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.075s user 0.000s sys 0.001s
I20250905 20:00:03.123287 30379 raft_consensus.cc:491] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:03.123647 30559 raft_consensus.cc:513] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:03.120554 30594 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
W20250905 20:00:03.123603 28051 connection.cc:232] RPC call timeout handler was delayed by -3.4973221010001225s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1607s for a response
I20250905 20:00:03.121690 30554 raft_consensus.cc:491] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:03.116847 30507 raft_consensus.cc:491] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:03.111604 30517 raft_consensus.cc:2747] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:03.134323 30196 raft_consensus.cc:513] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:03.142995 28051 connection.cc:232] RPC call timeout handler was delayed by -3.3893221010000616s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17270000000000002s for a response
I20250905 20:00:03.138859 28004 raft_consensus.cc:1238] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:03.133044 30393 raft_consensus.cc:3058] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:03.166916 30507 raft_consensus.cc:513] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:03.152256 28051 connection.cc:232] RPC call timeout handler was delayed by -3.3632221010000194s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1756s for a response
W20250905 20:00:03.152889 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 63865.2us
I20250905 20:00:03.119659 30471 raft_consensus.cc:513] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:03.156458 30559 leader_election.cc:290] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:03.159117 30196 leader_election.cc:290] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:03.137280 30240 raft_consensus.cc:2802] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 20:00:03.172271 30240 raft_consensus.cc:491] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:03.172688 30240 raft_consensus.cc:3058] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:03.172720 30471 leader_election.cc:290] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:03.174633 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 534863us
W20250905 20:00:03.177030 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 116007us
I20250905 20:00:03.177295 30588 raft_consensus.cc:2802] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 20:00:03.177863 30588 raft_consensus.cc:491] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:03.178122 27975 leader_election.cc:304] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 20:00:03.159735 30574 log.cc:927] Time spent T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.534s user 0.004s sys 0.000s
I20250905 20:00:03.149407 30379 raft_consensus.cc:513] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:03.180955 30379 leader_election.cc:290] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:03.160627 28004 raft_consensus.cc:1238] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:03.181218 30544 raft_consensus.cc:2802] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:03.181668 30544 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:03.182025 30544 raft_consensus.cc:3058] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:03.182698 28154 raft_consensus.cc:1273] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250905 20:00:03.156524 30442 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:03.184187 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 118964us
I20250905 20:00:03.185096 27973 leader_election.cc:304] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:03.180990 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:03.186002 30596 consensus_queue.cc:1035] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 20:00:03.187893 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:03.159960 30517 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:03.189401 30517 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:03.191685 30517 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:03.170781 28051 connection.cc:232] RPC call timeout handler was delayed by -3.2318221010000343s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1902s for a response
I20250905 20:00:03.171986 28001 raft_consensus.cc:2391] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
W20250905 20:00:03.192903 28051 connection.cc:232] RPC call timeout handler was delayed by -3.135522101000106s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2009s for a response
W20250905 20:00:03.193313 28051 connection.cc:232] RPC call timeout handler was delayed by -3.039222101000064s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2116s for a response
W20250905 20:00:03.193624 28051 connection.cc:232] RPC call timeout handler was delayed by -3.039222101000064s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2116s for a response
W20250905 20:00:03.193962 28051 connection.cc:232] RPC call timeout handler was delayed by -2.9654221010000583s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2198s for a response
W20250905 20:00:03.194231 28051 connection.cc:232] RPC call timeout handler was delayed by -2.7714066569999432s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2057s for a response
W20250905 20:00:03.194586 28051 connection.cc:232] RPC call timeout handler was delayed by -1.2662221010000394s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.40860000000000007s for a response
I20250905 20:00:03.195026 30602 consensus_queue.cc:1046] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250905 20:00:03.147949 30594 raft_consensus.cc:3058] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Advancing to term 8
W20250905 20:00:03.196632 28083 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1467) took 1492 ms (client timeout 1613 ms). Trace:
W20250905 20:00:03.197095 28083 rpcz_store.cc:269] 0905 20:00:01.704669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:01.704823 (+ 154us) service_pool.cc:225] Handling call
0905 20:00:03.196442 (+1491619us) spinlock_profiling.cc:243] Waited 794 ms on lock 0x7b540029ac54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:03.196575 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1826726784}
I20250905 20:00:03.139496 30600 raft_consensus.cc:2747] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:03.198690 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 20:00:03.200899 30568 log.cc:927] Time spent T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.551s user 0.000s sys 0.002s
I20250905 20:00:03.201951 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:03.195915 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:03.150372 30554 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:03.165217 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:03.163537 30374 leader_election.cc:290] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:03.207075 30017 leader_election.cc:290] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:03.206831 30319 raft_consensus.cc:513] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:03.209507 30319 leader_election.cc:290] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:03.206862 28752 raft_consensus.cc:2466] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
W20250905 20:00:03.189827 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:03.211853 28717 leader_election.cc:304] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:03.219337 28123 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:03.221506 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:03.223016 27973 leader_election.cc:304] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:03.224905 27973 leader_election.cc:304] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:03.227191 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:03.229549 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:03.231194 28002 raft_consensus.cc:1238] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:03.232956 30319 raft_consensus.cc:2747] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 20:00:03.234732 30263 raft_consensus.cc:2802] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:03.242621 30263 raft_consensus.cc:491] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:03.243253 30263 raft_consensus.cc:3058] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:03.238405 27973 leader_election.cc:304] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:03.239509 30470 raft_consensus.cc:2747] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:03.248616 30543 consensus_queue.cc:1035] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250905 20:00:03.252251 30470 raft_consensus.cc:2802] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election won for term 6
I20250905 20:00:03.253475 30470 raft_consensus.cc:695] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:03.255584 30470 consensus_queue.cc:237] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:03.258643 30572 consensus_queue.cc:1046] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250905 20:00:03.260919 30582 raft_consensus.cc:3053] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Stepping down as leader of term 4
I20250905 20:00:03.261552 30582 raft_consensus.cc:738] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:03.263356 30582 consensus_queue.cc:260] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:03.267853 30530 consensus_queue.cc:1035] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 20:00:03.269193 30530 consensus_queue.cc:1230] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer e4413ae5e0b64b52a6f360ac580239b9 log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20250905 20:00:03.178246 30588 raft_consensus.cc:3058] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
W20250905 20:00:03.223769 30545 consensus_meta.cc:287] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.721s user 0.006s sys 0.006s
I20250905 20:00:03.303880 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:03.170210 30507 leader_election.cc:290] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:03.365900 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:03.363503 30529 raft_consensus.cc:491] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:03.368901 30529 raft_consensus.cc:513] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:03.375747 30402 consensus_meta.cc:287] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.523s user 0.006s sys 0.004s
I20250905 20:00:03.376868 30402 raft_consensus.cc:513] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:03.380441 30586 raft_consensus.cc:513] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:03.284226 30374 raft_consensus.cc:2802] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:03.394268 30374 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:03.394963 30374 raft_consensus.cc:3058] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:03.291848 28051 leader_election.cc:304] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 20:00:03.439551 27973 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:03.447175 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:03.225879 30559 consensus_queue.cc:1035] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.136s
I20250905 20:00:03.510037 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:03.493458 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:03.449863 30471 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:03.512876 30196 raft_consensus.cc:2747] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250905 20:00:03.558521 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 327047us
I20250905 20:00:03.568787 28002 raft_consensus.cc:1238] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:03.575976 28154 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "0860997b3256400084f5884190bbe6e2"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
from {username='slave'} at 127.0.0.1:36164
I20250905 20:00:03.576978 28154 raft_consensus.cc:604] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Received request to transfer leadership to TS e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:03.577458 28154 raft_consensus.cc:612] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250905 20:00:03.583840 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 372039us
I20250905 20:00:03.585603 28717 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:03.448258 30545 raft_consensus.cc:513] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 20:00:03.589893 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:03.594867 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 314751us
I20250905 20:00:03.593494 30602 consensus_queue.cc:1046] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.004s
I20250905 20:00:03.597572 30602 consensus_queue.cc:1230] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer cd764bfeebac4cc9ad851cd4a6a7d3b0 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20250905 20:00:03.587646 30383 raft_consensus.cc:491] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:03.593200 30608 raft_consensus.cc:2802] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 20:00:03.599069 30608 raft_consensus.cc:491] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:03.599757 30608 raft_consensus.cc:3058] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:03.599913 30393 raft_consensus.cc:513] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:03.600659 30602 raft_consensus.cc:2747] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250905 20:00:03.602032 28717 leader_election.cc:304] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:03.598858 30383 raft_consensus.cc:513] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:03.603081 28153 raft_consensus.cc:1238] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: []
W20250905 20:00:03.607286 30609 log.cc:927] Time spent T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20250905 20:00:03.607983 30602 raft_consensus.cc:2747] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:03.609923 30393 leader_election.cc:290] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:03.611737 30383 leader_election.cc:290] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:03.610354 27974 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:03.621731 30383 raft_consensus.cc:2802] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250905 20:00:03.622299 30383 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:03.622804 30383 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Advancing to term 10
I20250905 20:00:03.624857 30545 leader_election.cc:290] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:03.612444 30539 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:03.626049 30539 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:03.626909 28003 raft_consensus.cc:2391] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:03.627872 30569 consensus_queue.cc:1046] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250905 20:00:03.629287 30602 raft_consensus.cc:3053] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 5 LEADER]: Stepping down as leader of term 5
I20250905 20:00:03.612617 30393 raft_consensus.cc:491] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:03.615376 30593 leader_election.cc:290] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:03.629683 30602 raft_consensus.cc:738] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:03.630175 30393 raft_consensus.cc:513] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:03.630589 30602 consensus_queue.cc:260] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:03.632582 30529 leader_election.cc:290] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:03.632643 30393 leader_election.cc:290] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:03.638839 30539 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:03.644385 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:03.650099 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:03.650978 28750 raft_consensus.cc:3058] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:03.650802 30608 raft_consensus.cc:513] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:03.653828 30608 leader_election.cc:290] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:03.655248 30402 leader_election.cc:290] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:03.657593 30586 leader_election.cc:290] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:03.554867 30471 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:03.667096 28001 raft_consensus.cc:2466] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 8.
W20250905 20:00:03.598224 30599 log.cc:927] Time spent T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20250905 20:00:03.685730 28750 raft_consensus.cc:2408] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250905 20:00:03.689157 28050 leader_election.cc:304] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
W20250905 20:00:03.682695 30240 consensus_meta.cc:287] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.509s user 0.006s sys 0.006s
W20250905 20:00:03.711962 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 348760us
I20250905 20:00:03.682978 30559 raft_consensus.cc:491] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:03.721741 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:03.723222 28152 raft_consensus.cc:1273] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20250905 20:00:03.833202 27973 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:03.962487 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:03.963460 28076 raft_consensus.cc:2466] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
W20250905 20:00:03.966707 28158 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.670s user 0.038s sys 0.050s
W20250905 20:00:03.970350 28158 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.674s user 0.038s sys 0.050s
W20250905 20:00:03.984182 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:03.988505 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:03.989411 28080 raft_consensus.cc:3058] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:03.991772 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:03.992578 28085 raft_consensus.cc:3058] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:03.709465 30196 raft_consensus.cc:2747] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 20:00:04.049610 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:04.050590 28078 raft_consensus.cc:3058] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
W20250905 20:00:04.057235 28150 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.676s user 0.040s sys 0.065s
I20250905 20:00:04.057971 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:03.686802 30383 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:03.719913 30240 raft_consensus.cc:513] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:04.355072 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:04.002513 30471 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:04.330129 30599 log.cc:927] Time spent T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.171s user 0.001s sys 0.000s
W20250905 20:00:04.102360 30263 consensus_meta.cc:287] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.858s user 0.011s sys 0.002s
I20250905 20:00:04.351186 30539 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
W20250905 20:00:04.236455 28715 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.970s user 0.039s sys 0.136s
I20250905 20:00:04.342938 30604 consensus_queue.cc:1046] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.102s
W20250905 20:00:04.352185 30148 consensus_meta.cc:287] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 1.245s user 0.010s sys 0.003s
I20250905 20:00:04.098141 28750 raft_consensus.cc:3053] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Stepping down as leader of term 4
I20250905 20:00:04.301023 28085 raft_consensus.cc:2466] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 7.
I20250905 20:00:04.299328 30442 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
W20250905 20:00:04.060209 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 358788us
W20250905 20:00:04.352180 30594 consensus_meta.cc:287] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.156s user 0.013s sys 0.000s
W20250905 20:00:04.236094 30544 consensus_meta.cc:287] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 1.054s user 0.013s sys 0.000s
W20250905 20:00:04.390600 30317 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.794s user 0.019s sys 0.024s
W20250905 20:00:04.391309 30459 consensus_meta.cc:287] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.526s user 0.014s sys 0.000s
W20250905 20:00:04.328338 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 487919us
W20250905 20:00:04.379384 27976 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.856s user 0.054s sys 0.103s
W20250905 20:00:04.329751 28122 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.753s user 0.007s sys 0.008s
W20250905 20:00:04.407574 30588 consensus_meta.cc:287] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 1.047s user 0.008s sys 0.004s
I20250905 20:00:04.199128 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:03.994571 30559 raft_consensus.cc:513] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:04.185266 27975 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.690s user 0.015s sys 0.073s
I20250905 20:00:04.445286 28750 raft_consensus.cc:738] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
W20250905 20:00:04.499203 27975 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.004s user 0.015s sys 0.104s
W20250905 20:00:04.440496 28150 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.079s user 0.058s sys 0.106s
I20250905 20:00:04.441390 28157 raft_consensus.cc:1238] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: []
W20250905 20:00:04.408771 28049 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.839s user 0.041s sys 0.077s
I20250905 20:00:04.136148 28078 raft_consensus.cc:2466] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:04.448686 30459 raft_consensus.cc:513] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:04.513864 28003 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 2019) took 2144 ms (client timeout 1650 ms). Trace:
W20250905 20:00:04.514552 28003 rpcz_store.cc:269] 0905 20:00:02.369679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:02.369837 (+ 158us) service_pool.cc:225] Handling call
0905 20:00:04.513567 (+2143730us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54000cb054. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:04.513773 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2856484096}
I20250905 20:00:04.515165 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:04.515848 28078 raft_consensus.cc:3058] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
W20250905 20:00:04.454974 28122 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.878s user 0.008s sys 0.015s
I20250905 20:00:04.516302 28008 raft_consensus.cc:1238] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250905 20:00:04.458182 28079 raft_consensus.cc:3058] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 5
W20250905 20:00:04.516650 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 940026us
I20250905 20:00:04.455833 28081 raft_consensus.cc:3058] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:04.443452 30419 raft_consensus.cc:3058] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:04.517517 28750 consensus_queue.cc:260] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:04.487334 28001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1445) took 1040 ms. Trace:
W20250905 20:00:04.420167 28125 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.849s user 0.018s sys 0.061s
W20250905 20:00:04.528949 28125 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.958s user 0.018s sys 0.061s
I20250905 20:00:04.497015 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 20:00:04.444523 28715 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.178s user 0.049s sys 0.165s
I20250905 20:00:04.532547 28752 raft_consensus.cc:2466] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
W20250905 20:00:04.532732 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 1.26676e+06us
W20250905 20:00:04.535349 28715 connection.cc:232] RPC call timeout handler was delayed by -1.2662776350000513s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1841s for a response
W20250905 20:00:04.536108 28715 connection.cc:232] RPC call timeout handler was delayed by -1.105734286000029s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1553s for a response
W20250905 20:00:04.541755 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:04.595898 28750 raft_consensus.cc:3058] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Advancing to term 5
W20250905 20:00:04.650830 28715 connection.cc:232] RPC call timeout handler was delayed by -1.0580159810000396s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17s for a response
I20250905 20:00:04.511848 28080 raft_consensus.cc:2466] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:04.444267 30537 raft_consensus.cc:491] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:04.651576 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:04.513767 30319 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.990s user 0.030s sys 0.072s
I20250905 20:00:04.664364 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:04.665230 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:04.666189 28080 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:04.445449 30618 raft_consensus.cc:3053] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Stepping down as leader of term 4
W20250905 20:00:04.496953 28158 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.522s user 0.019s sys 0.016s
I20250905 20:00:04.446482 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:04.688551 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:04.732882 28085 raft_consensus.cc:3058] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:04.442759 30240 leader_election.cc:290] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:04.528553 28001 rpcz_store.cc:276] 0905 20:00:03.447053 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:03.447172 (+ 119us) service_pool.cc:225] Handling call
0905 20:00:04.487146 (+1039974us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b54000ff5d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:04.487273 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":287714176}
I20250905 20:00:04.734751 28082 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1517) took 1003 ms. Trace:
I20250905 20:00:04.679769 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:04.735584 28082 rpcz_store.cc:276] 0905 20:00:03.731248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:03.731719 (+ 471us) service_pool.cc:225] Handling call
0905 20:00:04.688311 (+956592us) spinlock_profiling.cc:243] Waited 88.5 ms on lock 0x7b5800204f98. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b7db60 00007fe4a2b7d444 00007fe49c2050c4 00007fe49c205052
0905 20:00:04.734678 (+ 46367us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":203640064}
I20250905 20:00:04.736765 28154 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:36164
W20250905 20:00:04.659948 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:04.596078 30612 raft_consensus.cc:491] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
W20250905 20:00:04.427532 30571 log.cc:927] Time spent T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410: Append to log took a long time: real 1.900s user 0.006s sys 0.015s
I20250905 20:00:04.780973 28081 raft_consensus.cc:2466] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:04.780642 30612 raft_consensus.cc:513] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:04.783717 30612 leader_election.cc:290] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:04.595789 30393 raft_consensus.cc:491] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:04.666101 30537 raft_consensus.cc:513] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:04.786957 30393 raft_consensus.cc:513] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:04.789260 30537 leader_election.cc:290] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:04.662803 28078 raft_consensus.cc:2466] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:04.472875 30559 leader_election.cc:290] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:04.778194 28085 raft_consensus.cc:2466] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
W20250905 20:00:04.448055 30317 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.851s user 0.019s sys 0.027s
I20250905 20:00:04.535442 30552 raft_consensus.cc:491] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:04.450093 30383 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 10 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:04.484472 30582 raft_consensus.cc:3058] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:04.454310 30624 raft_consensus.cc:491] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:04.449620 30619 consensus_queue.cc:1035] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.232s
I20250905 20:00:04.447494 30594 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:04.447650 30544 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:04.444252 30263 raft_consensus.cc:513] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:04.487282 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:04.522285 30517 raft_consensus.cc:491] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:04.734671 30543 raft_consensus.cc:491] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
W20250905 20:00:04.495268 30374 consensus_meta.cc:287] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.100s user 0.016s sys 0.000s
W20250905 20:00:04.454947 27976 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.950s user 0.056s sys 0.105s
I20250905 20:00:04.445194 30148 raft_consensus.cc:513] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:04.447707 28051 leader_election.cc:304] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 20:00:04.511633 30554 consensus_meta.cc:287] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.308s user 0.011s sys 0.003s
I20250905 20:00:04.637977 30610 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
W20250905 20:00:04.738103 28715 leader_election.cc:336] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.553s (SENT)
I20250905 20:00:04.443732 28006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1438) took 1912 ms. Trace:
W20250905 20:00:04.688898 28122 leader_election.cc:336] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.703s (SENT)
I20250905 20:00:04.684515 30402 raft_consensus.cc:491] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:04.817664 28715 leader_election.cc:304] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:04.444209 30539 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:04.818564 28153 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2117) took 1639 ms. Trace:
I20250905 20:00:04.733841 30618 raft_consensus.cc:738] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:04.450882 30569 consensus_queue.cc:1046] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.304s
W20250905 20:00:04.664481 30319 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.141s user 0.034s sys 0.077s
I20250905 20:00:04.753549 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:04.446416 30442 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:04.756619 30176 raft_consensus.cc:491] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:04.790884 30552 raft_consensus.cc:513] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:04.757299 28157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2123) took 1035 ms. Trace:
I20250905 20:00:04.755770 30459 leader_election.cc:290] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:04.758903 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:04.768424 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:04.769819 28010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1442) took 2185 ms. Trace:
I20250905 20:00:04.847626 28083 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 9.
I20250905 20:00:04.848234 28083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1512) took 1129 ms. Trace:
I20250905 20:00:04.848661 28083 rpcz_store.cc:276] 0905 20:00:03.718943 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:03.719119 (+ 176us) service_pool.cc:225] Handling call
0905 20:00:04.768234 (+1049115us) spinlock_profiling.cc:243] Waited 97 ms on lock 0x7b5800204f98. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b7db60 00007fe4a2b7d444 00007fe49c2050c4 00007fe49c205052
0905 20:00:04.848165 (+ 79931us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":223179648}
I20250905 20:00:04.774322 27974 leader_election.cc:304] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 20:00:04.826467 30615 log.cc:927] Time spent T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.428s user 0.000s sys 0.001s
I20250905 20:00:04.777529 30586 consensus_queue.cc:1046] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 20:00:04.847541 28010 rpcz_store.cc:276] 0905 20:00:02.584021 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:02.683848 (+ 99827us) service_pool.cc:225] Handling call
0905 20:00:03.181561 (+497713us) raft_consensus.cc:1469] Updating replica for 2 ops
0905 20:00:03.181949 (+ 388us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:03.181979 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0905 20:00:03.182574 (+ 595us) log.cc:844] Serialized 52 byte log entry
0905 20:00:03.545884 (+363310us) raft_consensus.cc:1643] Marking committed up to 2
0905 20:00:03.545930 (+ 46us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:03.546034 (+ 104us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:04.352458 (+806424us) raft_consensus.cc:1683] finished
0905 20:00:04.352484 (+ 26us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:04.769502 (+417018us) spinlock_profiling.cc:243] Waited 497 ms on lock 0x7b54003412d0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:04.769742 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":829,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89,"spinlock_wait_cycles":1143620352,"thread_start_us":362824,"threads_started":1,"wal-append.queue_time_us":781842}
W20250905 20:00:04.849798 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 75567.5us
I20250905 20:00:04.823134 30627 raft_consensus.cc:491] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:04.803822 28080 raft_consensus.cc:2466] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 8.
I20250905 20:00:04.805977 28750 raft_consensus.cc:1273] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250905 20:00:04.482820 30501 raft_consensus.cc:491] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:04.807778 28746 raft_consensus.cc:2466] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 20:00:04.809587 30544 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:04.804771 30383 raft_consensus.cc:491] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:04.726776 28079 raft_consensus.cc:2466] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:04.817778 28006 rpcz_store.cc:276] 0905 20:00:02.531410 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:02.598909 (+ 67499us) service_pool.cc:225] Handling call
0905 20:00:02.599292 (+ 383us) raft_consensus.cc:1469] Updating replica for 2 ops
0905 20:00:02.617431 (+ 18139us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:02.617473 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0905 20:00:02.630746 (+ 13273us) log.cc:844] Serialized 52 byte log entry
0905 20:00:03.052452 (+421706us) raft_consensus.cc:1643] Marking committed up to 2
0905 20:00:03.121642 (+ 69190us) log.cc:844] Serialized 14 byte log entry
0905 20:00:03.183625 (+ 61983us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:03.183698 (+ 73us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:04.443463 (+1259765us) raft_consensus.cc:1683] finished
0905 20:00:04.443487 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:04.443664 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":82603,"prepare.queue_time_us":155554,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":83426,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":20009,"thread_start_us":400070,"threads_started":2,"wal-append.queue_time_us":683709}
I20250905 20:00:04.818140 28122 leader_election.cc:304] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
W20250905 20:00:04.925231 30591 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20250905 20:00:04.455264 30588 raft_consensus.cc:513] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:04.835364 28157 rpcz_store.cc:276] 0905 20:00:03.721400 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:03.722206 (+ 806us) service_pool.cc:225] Handling call
0905 20:00:03.722592 (+ 386us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:04.502285 (+779693us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:04.756962 (+254677us) spinlock_profiling.cc:243] Waited 718 ms on lock 0x7b540036e7d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:04.757216 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1652465408}
W20250905 20:00:04.838032 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 390370us
W20250905 20:00:04.838263 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 100195us
I20250905 20:00:04.789426 30393 leader_election.cc:290] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:04.505730 28049 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.936s user 0.041s sys 0.077s
I20250905 20:00:04.815656 30628 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:04.994149 28715 leader_election.cc:336] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.619s (SENT)
W20250905 20:00:04.994504 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:04.995186 28715 leader_election.cc:336] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.630s (SENT)
I20250905 20:00:04.995718 28715 leader_election.cc:304] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:04.846788 30585 raft_consensus.cc:491] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:04.813213 30517 raft_consensus.cc:513] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:05.009392 28715 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.700s (SENT) [suppressed 45 similar messages]
W20250905 20:00:05.010373 28715 leader_election.cc:336] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.763s (SENT)
I20250905 20:00:05.010921 28715 leader_election.cc:304] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:04.830914 30559 raft_consensus.cc:2747] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:04.801762 30624 raft_consensus.cc:513] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:04.924968 28123 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.008s sys 0.021s
I20250905 20:00:04.951786 30529 raft_consensus.cc:491] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
W20250905 20:00:04.900827 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:05.013944 28123 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.594s user 0.009s sys 0.021s
I20250905 20:00:04.863615 28750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40904 (request call id 623) took 1152 ms. Trace:
I20250905 20:00:04.928184 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:04.851686 30569 consensus_queue.cc:1230] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer cd764bfeebac4cc9ad851cd4a6a7d3b0 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20250905 20:00:04.849016 30604 raft_consensus.cc:2747] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:04.819057 30374 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:04.938408 27973 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.624s user 0.023s sys 0.008s
W20250905 20:00:05.021970 27973 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.708s user 0.024s sys 0.008s
I20250905 20:00:04.815593 30543 raft_consensus.cc:513] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:04.931610 28048 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.692s user 0.039s sys 0.052s
I20250905 20:00:04.856877 28744 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Advancing to term 10
I20250905 20:00:05.018863 30451 raft_consensus.cc:2747] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 20:00:04.976202 30618 consensus_queue.cc:260] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:05.033213 30618 raft_consensus.cc:3058] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:05.033311 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:04.952548 28008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1434) took 2410 ms. Trace:
I20250905 20:00:04.985095 30501 raft_consensus.cc:513] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:05.034849 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:05.036585 30501 leader_election.cc:290] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:04.997923 30383 raft_consensus.cc:513] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:05.034652 28008 rpcz_store.cc:276] 0905 20:00:02.524613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:02.555588 (+ 30975us) service_pool.cc:225] Handling call
0905 20:00:04.515939 (+1960351us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:04.516834 (+ 895us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:04.934853 (+418019us) spinlock_profiling.cc:243] Waited 1.95 s on lock 0x7b54000cb050. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:04.935081 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4489420032}
I20250905 20:00:05.039561 30383 leader_election.cc:290] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:05.041730 30624 leader_election.cc:290] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:04.828104 30610 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:05.043661 28744 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 10.
I20250905 20:00:05.044901 30610 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:05.013352 30585 raft_consensus.cc:513] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:05.045608 30610 raft_consensus.cc:491] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:05.047091 30634 log.cc:927] Time spent T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
W20250905 20:00:04.955441 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 423640us
W20250905 20:00:04.986660 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 297764us
W20250905 20:00:05.051038 28122 leader_election.cc:336] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.713s (SENT)
I20250905 20:00:05.051697 28122 leader_election.cc:304] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:04.822191 28153 rpcz_store.cc:276] 0905 20:00:03.179585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:03.181110 (+ 1525us) service_pool.cc:225] Handling call
0905 20:00:03.181391 (+ 281us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:03.603531 (+422140us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:04.818200 (+1214669us) spinlock_profiling.cc:243] Waited 421 ms on lock 0x7b54000ed3d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:04.818483 (+ 283us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":969302912}
I20250905 20:00:04.865870 30633 raft_consensus.cc:491] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:04.925508 30263 leader_election.cc:290] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:04.825959 30554 raft_consensus.cc:513] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:04.992259 28124 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.038s sys 0.045s
W20250905 20:00:05.095361 28124 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.803s user 0.038s sys 0.045s
W20250905 20:00:04.952320 28077 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.606s user 0.021s sys 0.009s
I20250905 20:00:04.849330 30176 raft_consensus.cc:513] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:05.000775 28150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1578) took 1775 ms. Trace:
I20250905 20:00:04.866609 28154 raft_consensus.cc:604] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Received request to transfer leadership to TS cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:05.102896 28154 raft_consensus.cc:612] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250905 20:00:05.016464 28123 leader_election.cc:304] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:05.017927 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:05.014515 28750 rpcz_store.cc:276] 0905 20:00:03.711172 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:03.711404 (+ 232us) service_pool.cc:225] Handling call
0905 20:00:03.711770 (+ 366us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:04.863201 (+1151431us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:04.863541 (+ 340us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 20:00:05.001490 30420 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.013s sys 0.021s
W20250905 20:00:05.107314 30420 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.659s user 0.014s sys 0.021s
I20250905 20:00:05.108004 30420 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:05.108901 30176 leader_election.cc:290] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:05.108701 30420 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:05.101325 28077 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.755s user 0.022s sys 0.010s
W20250905 20:00:05.024406 28048 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.817s user 0.039s sys 0.052s
W20250905 20:00:05.112282 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:05.113719 28084 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.695s user 0.019s sys 0.101s
W20250905 20:00:05.029731 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:05.031354 27973 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:05.114133 28084 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.697s user 0.019s sys 0.101s
W20250905 20:00:04.999897 30470 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s user 0.042s sys 0.058s
I20250905 20:00:05.007496 30507 raft_consensus.cc:491] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:05.021581 30606 raft_consensus.cc:2747] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 20:00:04.826238 30402 raft_consensus.cc:513] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:05.047237 30585 leader_election.cc:290] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:04.864992 30540 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.021s sys 0.035s
W20250905 20:00:05.134797 30540 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.799s user 0.022s sys 0.036s
I20250905 20:00:04.854668 30530 raft_consensus.cc:2802] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 20:00:05.135267 30530 raft_consensus.cc:491] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:05.135545 28001 raft_consensus.cc:1238] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:05.135679 30530 raft_consensus.cc:3058] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 2 FOLLOWER]: Advancing to term 3
I20250905 20:00:05.056130 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:05.136703 28083 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Advancing to term 10
W20250905 20:00:05.064898 28122 leader_election.cc:336] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.815s (SENT)
W20250905 20:00:05.043449 30419 consensus_meta.cc:287] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.508s user 0.010s sys 0.001s
I20250905 20:00:05.143006 30552 leader_election.cc:290] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:05.143527 30419 consensus_queue.cc:1046] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 20:00:05.143931 30552 raft_consensus.cc:2802] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 20:00:05.070417 28051 leader_election.cc:304] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:05.144503 30552 raft_consensus.cc:491] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:05.145088 30552 raft_consensus.cc:3058] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
W20250905 20:00:05.145337 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 124275us
W20250905 20:00:05.145964 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:05.146692 28051 leader_election.cc:304] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:05.148829 28051 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:05.150107 28159 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 20:00:05.151273 30579 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
W20250905 20:00:05.152488 28159 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39586 (request call id 292) took 2645 ms (client timeout 2142 ms). Trace:
W20250905 20:00:05.153126 28159 rpcz_store.cc:269] 0905 20:00:02.506984 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:02.507128 (+ 144us) service_pool.cc:225] Handling call
0905 20:00:05.152137 (+2645009us) spinlock_profiling.cc:243] Waited 2.62 s on lock 0x7b5400073ad4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:05.152344 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6017327872}
I20250905 20:00:05.073493 28155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1558) took 2847 ms. Trace:
I20250905 20:00:05.154281 28155 rpcz_store.cc:276] 0905 20:00:02.202910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:02.203043 (+ 133us) service_pool.cc:225] Handling call
0905 20:00:02.203388 (+ 345us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:02.349820 (+146432us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:02.349859 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:02.356400 (+ 6541us) log.cc:844] Serialized 26 byte log entry
0905 20:00:03.195972 (+839572us) spinlock_profiling.cc:243] Waited 15.9 ms on lock 0x7b180039e3e0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49ac86d47 00007fe4a38dcb99 00007fe49aca369b 00007fe49aca193f 00007fe49aca3c02 00007fe49d8572bb 00007fe49d85cdc3 00007fe49d85d171 00007fe49dbec09f
0905 20:00:03.552273 (+356301us) raft_consensus.cc:1643] Marking committed up to 3
0905 20:00:03.552335 (+ 62us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:04.452208 (+899873us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b5400177354. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:04.452289 (+ 81us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:04.773142 (+320853us) spinlock_profiling.cc:243] Waited 44.5 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49d85891e 00007fe49d857c66 00007fe49d857888 00007fe49d8612c1 00007fe49d861279 00007fe49d861209
0905 20:00:05.050244 (+277102us) raft_consensus.cc:1683] finished
0905 20:00:05.050280 (+ 36us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:05.050487 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2877,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":31164,"spinlock_wait_cycles":474728064,"thread_start_us":356057,"threads_started":1,"wal-append.queue_time_us":1800571}
W20250905 20:00:05.024370 28715 leader_election.cc:336] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.841s (SENT)
I20250905 20:00:05.156195 28715 leader_election.cc:304] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
W20250905 20:00:05.157152 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 132803us
W20250905 20:00:05.157927 28715 leader_election.cc:336] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.911s (SENT)
I20250905 20:00:05.158401 30600 raft_consensus.cc:2747] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:05.159940 30543 leader_election.cc:290] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:04.868892 30627 raft_consensus.cc:513] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:05.117133 30470 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.636s user 0.043s sys 0.059s
I20250905 20:00:05.162863 30627 leader_election.cc:290] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:05.164640 30402 leader_election.cc:290] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:05.164736 30588 leader_election.cc:290] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:05.013540 30628 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:05.165814 30557 raft_consensus.cc:3053] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Stepping down as leader of term 4
W20250905 20:00:05.143069 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 78235.7us
W20250905 20:00:05.167457 28122 leader_election.cc:336] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 2.028s (SENT)
I20250905 20:00:05.167974 28122 leader_election.cc:304] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:05.101817 28150 rpcz_store.cc:276] 0905 20:00:03.225586 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:03.225781 (+ 195us) service_pool.cc:225] Handling call
0905 20:00:03.286344 (+ 60563us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:03.286673 (+ 329us) raft_consensus.cc:1532] Early marking committed up to index 7
0905 20:00:04.501388 (+1214715us) log.cc:844] Serialized 14 byte log entry
0905 20:00:04.978381 (+476993us) log.cc:844] Serialized 14 byte log entry
0905 20:00:05.000045 (+ 21664us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:05.000172 (+ 127us) raft_consensus.cc:1643] Marking committed up to 7
0905 20:00:05.000323 (+ 151us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:05.000450 (+ 127us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:05.000657 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23068,"prepare.queue_time_us":1215284,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":28,"raft.queue_time_us":37517,"raft.run_wall_time_us":9789,"spinlock_wait_cycles":78650752,"thread_start_us":1614636,"threads_started":2}
I20250905 20:00:05.117070 30379 raft_consensus.cc:491] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:05.126312 28010 raft_consensus.cc:1238] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:05.044914 30517 leader_election.cc:290] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:05.158308 28715 leader_election.cc:304] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:05.126799 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:05.170677 28085 raft_consensus.cc:3058] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:05.171927 28010 raft_consensus.cc:1238] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:05.172353 27973 leader_election.cc:304] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:05.127408 30507 raft_consensus.cc:513] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:05.128350 30559 consensus_queue.cc:1046] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 20:00:05.131639 30641 raft_consensus.cc:2747] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:05.143795 30530 raft_consensus.cc:513] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:05.152069 30579 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:05.125015 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:05.185184 28080 raft_consensus.cc:3058] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:05.185349 28083 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 10.
I20250905 20:00:05.126582 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:05.175637 28122 leader_election.cc:336] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 2.155s (SENT)
I20250905 20:00:05.028079 30529 raft_consensus.cc:513] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:05.166247 30557 raft_consensus.cc:738] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
W20250905 20:00:05.173544 30625 log.cc:927] Time spent T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.801s user 0.000s sys 0.002s
I20250905 20:00:05.134003 30645 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:05.186666 28122 leader_election.cc:304] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:05.188381 30619 raft_consensus.cc:2747] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250905 20:00:05.188853 30635 raft_consensus.cc:2747] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 20:00:05.189481 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:05.184068 30629 raft_consensus.cc:2747] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250905 20:00:05.068706 30610 raft_consensus.cc:513] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:05.209635 28715 leader_election.cc:336] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 2.641s (SENT)
I20250905 20:00:05.190714 30543 raft_consensus.cc:2802] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 20:00:05.177700 30148 leader_election.cc:290] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:05.097540 30633 raft_consensus.cc:513] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:05.123435 30594 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:05.264212 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 165882us
I20250905 20:00:05.265442 28125 leader_election.cc:304] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:05.267990 28125 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:05.204823 28085 raft_consensus.cc:2466] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
I20250905 20:00:05.206316 28158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1574) took 1988 ms. Trace:
I20250905 20:00:05.211076 30543 raft_consensus.cc:491] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:05.271054 28158 rpcz_store.cc:276] 0905 20:00:03.217447 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:03.217747 (+ 300us) service_pool.cc:225] Handling call
0905 20:00:03.218145 (+ 398us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:03.218421 (+ 276us) raft_consensus.cc:1532] Early marking committed up to index 3
0905 20:00:03.974417 (+755996us) log.cc:844] Serialized 14 byte log entry
0905 20:00:05.205864 (+1231447us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:05.205926 (+ 62us) raft_consensus.cc:1643] Marking committed up to 3
0905 20:00:05.205973 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:05.206108 (+ 135us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:05.206254 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1276016,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":42,"raft.queue_time_us":58171,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":94,"thread_start_us":1908872,"threads_started":3,"wal-append.queue_time_us":880421}
I20250905 20:00:05.197515 28081 pending_rounds.cc:85] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9: Aborting all ops after (but not including) 3
I20250905 20:00:05.273025 28081 pending_rounds.cc:107] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20250905 20:00:05.273417 28081 raft_consensus.cc:2887] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250905 20:00:05.190318 30628 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:05.191630 27973 leader_election.cc:304] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:05.209782 30442 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:05.208145 30537 raft_consensus.cc:2747] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:05.214179 28008 raft_consensus.cc:2466] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 3.
I20250905 20:00:05.181859 30517 raft_consensus.cc:491] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:05.182338 30591 raft_consensus.cc:2747] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250905 20:00:05.213614 28715 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
W20250905 20:00:05.261210 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 85610.6us
I20250905 20:00:05.209715 30539 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:05.212747 30641 raft_consensus.cc:2747] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250905 20:00:05.276589 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 85043.3us
I20250905 20:00:05.182483 30379 raft_consensus.cc:513] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:05.281324 30579 leader_election.cc:290] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:05.282191 30148 raft_consensus.cc:2802] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader election won for term 7
I20250905 20:00:05.282341 30557 consensus_queue.cc:260] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:05.286576 30594 raft_consensus.cc:2747] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 20:00:05.286775 28076 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1521) took 1301 ms. Trace:
I20250905 20:00:05.287487 30420 leader_election.cc:290] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:05.288834 30637 raft_consensus.cc:491] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:05.275138 30543 raft_consensus.cc:3058] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:05.297114 30603 raft_consensus.cc:2747] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:05.297413 30651 raft_consensus.cc:2802] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250905 20:00:05.298355 30650 raft_consensus.cc:491] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:05.299199 30652 raft_consensus.cc:491] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:05.302387 28080 raft_consensus.cc:2466] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
W20250905 20:00:05.313230 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 296850us
I20250905 20:00:05.316361 28151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1577) took 2091 ms. Trace:
W20250905 20:00:05.323498 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 174834us
I20250905 20:00:05.376848 28051 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
W20250905 20:00:05.325265 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:05.325459 27973 leader_election.cc:304] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:05.341118 28082 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1524) took 1175 ms. Trace:
I20250905 20:00:05.341156 28076 rpcz_store.cc:276] 0905 20:00:03.985283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:03.985466 (+ 183us) service_pool.cc:225] Handling call
0905 20:00:04.070864 (+ 85398us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:04.206914 (+136050us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:04.206962 (+ 48us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:04.514587 (+307625us) log.cc:844] Serialized 26 byte log entry
0905 20:00:05.166394 (+651807us) raft_consensus.cc:1643] Marking committed up to 2
0905 20:00:05.167753 (+ 1359us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:05.167843 (+ 90us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:05.286507 (+118664us) raft_consensus.cc:1683] finished
0905 20:00:05.286530 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:05.286706 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34121,"raft.queue_time_us":150095,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":156,"thread_start_us":460647,"threads_started":1,"wal-append.queue_time_us":626791}
W20250905 20:00:05.380977 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 55633.8us
W20250905 20:00:05.344363 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 134757us
I20250905 20:00:05.381873 27973 leader_election.cc:304] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
W20250905 20:00:05.382704 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:05.383320 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:05.383701 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:05.258534 30635 consensus_queue.cc:1046] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.068s
I20250905 20:00:05.210038 30645 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:05.334182 30442 raft_consensus.cc:2747] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250905 20:00:05.361003 30148 raft_consensus.cc:695] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:05.383723 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:05.383328 27973 leader_election.cc:304] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:05.401934 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:05.375474 28151 rpcz_store.cc:276] 0905 20:00:03.225059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:03.225234 (+ 175us) service_pool.cc:225] Handling call
0905 20:00:03.288687 (+ 63453us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:03.289298 (+ 611us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:03.289342 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:03.296757 (+ 7415us) log.cc:844] Serialized 26 byte log entry
0905 20:00:03.852021 (+555264us) raft_consensus.cc:1643] Marking committed up to 2
0905 20:00:03.852093 (+ 72us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:03.852203 (+ 110us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:05.153498 (+1301295us) spinlock_profiling.cc:243] Waited 32.7 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49d85891e 00007fe49d857c66 00007fe49d857888 00007fe49d8612c1 00007fe49d861279 00007fe49d861209
0905 20:00:05.277443 (+123945us) raft_consensus.cc:1683] finished
0905 20:00:05.277465 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:05.316294 (+ 38829us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":143,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":51699,"spinlock_wait_cycles":75298048,"thread_start_us":406466,"threads_started":1,"wal-append.queue_time_us":991028}
I20250905 20:00:05.380532 28082 rpcz_store.cc:276] 0905 20:00:04.165865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:04.752153 (+586288us) service_pool.cc:225] Handling call
0905 20:00:04.752496 (+ 343us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:04.752704 (+ 208us) raft_consensus.cc:1532] Early marking committed up to index 3
0905 20:00:05.171292 (+418588us) log.cc:844] Serialized 14 byte log entry
0905 20:00:05.340565 (+169273us) log.cc:844] Serialized 14 byte log entry
0905 20:00:05.340750 (+ 185us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:05.340810 (+ 60us) raft_consensus.cc:1643] Marking committed up to 3
0905 20:00:05.340856 (+ 46us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:05.340922 (+ 66us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:05.341056 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":535569,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":95,"raft.queue_time_us":419,"spinlock_wait_cycles":8442112,"thread_start_us":583558,"threads_started":2}
I20250905 20:00:05.384095 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:05.363658 30374 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:05.360584 30420 raft_consensus.cc:491] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:05.363092 30557 raft_consensus.cc:3058] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:05.427700 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:05.363276 30651 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250905 20:00:05.429963 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 46699.9us
I20250905 20:00:05.430702 28750 raft_consensus.cc:2391] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 8: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 20:00:05.322618 30517 raft_consensus.cc:513] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:05.435272 28749 raft_consensus.cc:1238] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:05.435271 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:05.438915 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:05.525981 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:05.562655 28749 raft_consensus.cc:1273] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250905 20:00:05.611413 28751 raft_consensus.cc:3058] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Advancing to term 7
W20250905 20:00:05.611847 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:05.542619 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:05.548096 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:05.612838 28748 raft_consensus.cc:2466] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:05.463615 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:05.536101 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:05.581876 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:05.580823 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 20:00:05.462008 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 295062us
I20250905 20:00:05.613670 28155 raft_consensus.cc:1811] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Handling vote request from an unknown peer cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:05.610677 27973 leader_election.cc:304] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:05.615600 28155 raft_consensus.cc:2408] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
W20250905 20:00:05.599442 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 222704us
W20250905 20:00:05.616683 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:05.612061 28159 raft_consensus.cc:2466] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 20:00:05.405782 30470 raft_consensus.cc:2802] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election won for term 5
I20250905 20:00:05.408949 30539 raft_consensus.cc:2802] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election won for term 5
I20250905 20:00:05.633163 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:05.634863 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:05.635555 28155 raft_consensus.cc:2466] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6.
I20250905 20:00:05.624630 28123 leader_election.cc:304] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:05.397068 30628 raft_consensus.cc:491] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:05.396382 30594 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:05.636796 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:05.549769 30640 log.cc:927] Time spent T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
W20250905 20:00:05.562654 30638 log.cc:927] Time spent T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.246s user 0.001s sys 0.000s
I20250905 20:00:05.640488 30470 raft_consensus.cc:695] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:05.620342 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:05.641093 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:05.436384 30263 raft_consensus.cc:2747] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 20:00:05.653069 30402 raft_consensus.cc:2802] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election won for term 5
I20250905 20:00:05.364226 30637 raft_consensus.cc:513] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:05.682526 30402 raft_consensus.cc:695] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:05.375908 30650 raft_consensus.cc:513] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:05.577641 30529 leader_election.cc:290] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:05.464344 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:05.537218 30374 raft_consensus.cc:491] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:05.572530 30540 raft_consensus.cc:491] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:05.465833 30651 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Advancing to term 9
I20250905 20:00:05.575088 30507 leader_election.cc:290] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:05.563416 30530 leader_election.cc:290] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:05.422567 30652 raft_consensus.cc:513] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:05.651417 30661 consensus_queue.cc:1046] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250905 20:00:05.639124 30633 leader_election.cc:290] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:05.596666 30610 leader_election.cc:290] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:05.655315 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:05.716488 28155 raft_consensus.cc:3058] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:05.649752 27973 leader_election.cc:304] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:05.665102 28159 raft_consensus.cc:2466] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 20:00:05.549172 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:05.419709 30554 leader_election.cc:290] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:05.670644 28717 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
W20250905 20:00:05.722267 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 51715.9us
W20250905 20:00:05.658630 30658 log.cc:927] Time spent T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.095s user 0.000s sys 0.001s
I20250905 20:00:05.666512 30594 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:05.575876 30559 raft_consensus.cc:2802] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 20:00:05.686851 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:05.688747 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:05.734447 28151 raft_consensus.cc:2391] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 4: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:05.690594 28051 leader_election.cc:304] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:05.669346 30240 raft_consensus.cc:2802] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:05.735179 30470 consensus_queue.cc:237] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:05.735482 30240 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:05.636672 30148 consensus_queue.cc:237] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:05.718134 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 68484.3us
W20250905 20:00:05.693892 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 60510.5us
I20250905 20:00:05.666981 30628 raft_consensus.cc:513] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:05.707023 28744 raft_consensus.cc:2466] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 20:00:05.536854 30420 raft_consensus.cc:513] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:05.690361 30588 raft_consensus.cc:491] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:05.691979 30379 leader_election.cc:290] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:05.733155 30559 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:05.690758 30663 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:05.769526 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 93127.5us
I20250905 20:00:05.652774 30539 raft_consensus.cc:695] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:05.711416 30540 raft_consensus.cc:513] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:05.739006 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 20:00:05.739538 30643 log.cc:927] Time spent T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.272s user 0.000s sys 0.001s
W20250905 20:00:05.749588 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:05.728262 30507 consensus_queue.cc:1035] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20250905 20:00:05.745218 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:05.784502 28153 raft_consensus.cc:3058] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:05.744720 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:05.788582 28151 raft_consensus.cc:2391] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 6: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:05.728128 30529 raft_consensus.cc:2802] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election won for term 5
I20250905 20:00:05.748222 27973 leader_election.cc:304] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:05.690719 28084 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1519) took 1958 ms. Trace:
I20250905 20:00:05.711781 30374 raft_consensus.cc:513] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:05.746901 30240 raft_consensus.cc:3058] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:05.740836 28077 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1513) took 2021 ms. Trace:
W20250905 20:00:05.694459 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:05.854936 28084 rpcz_store.cc:276] 0905 20:00:03.732358 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:03.732512 (+ 154us) service_pool.cc:225] Handling call
0905 20:00:03.990275 (+257763us) raft_consensus.cc:1469] Updating replica for 2 ops
0905 20:00:04.021869 (+ 31594us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:04.021915 (+ 46us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0905 20:00:04.386831 (+364916us) log.cc:844] Serialized 52 byte log entry
0905 20:00:05.114612 (+727781us) raft_consensus.cc:1643] Marking committed up to 3
0905 20:00:05.354375 (+239763us) log.cc:844] Serialized 14 byte log entry
0905 20:00:05.690319 (+335944us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:05.690397 (+ 78us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:05.690489 (+ 92us) raft_consensus.cc:1683] finished
0905 20:00:05.690500 (+ 11us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:05.690663 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3011,"prepare.queue_time_us":173748,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":33,"raft.queue_time_us":456299,"raft.run_cpu_time_us":374,"raft.run_wall_time_us":33109,"thread_start_us":1217097,"threads_started":3,"wal-append.queue_time_us":872732}
I20250905 20:00:05.863732 28123 leader_election.cc:304] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:05.731256 30442 raft_consensus.cc:2802] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:05.761838 30420 leader_election.cc:290] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:05.727509 30645 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:05.789022 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:05.757323 28743 raft_consensus.cc:2366] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 4. Current term is 5.
I20250905 20:00:05.764739 28745 raft_consensus.cc:3053] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Stepping down as leader of term 4
I20250905 20:00:05.828467 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:05.757929 30591 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:05.777117 30559 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:05.830879 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:05.783020 30661 raft_consensus.cc:2747] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250905 20:00:05.847901 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:05.780599 30517 leader_election.cc:290] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:05.787823 30667 raft_consensus.cc:3053] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 LEADER]: Stepping down as leader of term 4
I20250905 20:00:05.840837 28153 raft_consensus.cc:2466] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 20:00:05.801133 30530 raft_consensus.cc:491] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:05.884266 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:05.767915 30588 raft_consensus.cc:513] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:05.890048 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 117591us
I20250905 20:00:05.846632 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:05.751396 30402 consensus_queue.cc:237] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:05.780941 30663 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:05.905259 28151 raft_consensus.cc:2466] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 20:00:05.906649 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:05.907845 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:05.909199 28717 leader_election.cc:304] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:05.889379 30442 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:05.892231 30637 leader_election.cc:290] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:05.895325 30668 raft_consensus.cc:2802] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:05.921523 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:05.928506 28159 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 20:00:05.917460 28153 raft_consensus.cc:2466] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 8.
W20250905 20:00:05.922554 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 174402us
W20250905 20:00:05.925382 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:05.930322 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:05.897722 28745 raft_consensus.cc:738] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:05.931973 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:05.936898 27973 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:05.938078 28749 raft_consensus.cc:2391] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 5: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 20:00:05.958041 28156 raft_consensus.cc:3058] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
W20250905 20:00:05.958318 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:05.963104 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 112490us
I20250905 20:00:05.911803 30667 raft_consensus.cc:738] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
W20250905 20:00:05.966627 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:05.923553 30442 raft_consensus.cc:3058] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:05.980293 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:05.981580 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:05.983142 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:05.984283 30540 leader_election.cc:290] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:05.984571 28151 raft_consensus.cc:2408] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250905 20:00:05.986656 30540 raft_consensus.cc:2802] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 20:00:05.977006 28751 raft_consensus.cc:2466] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
I20250905 20:00:05.991253 28123 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:05.981554 30529 raft_consensus.cc:695] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:05.987574 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:05.995044 28151 raft_consensus.cc:2391] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 8: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:06.000468 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:06.001188 28153 raft_consensus.cc:2466] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:05.981026 28159 raft_consensus.cc:2466] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
I20250905 20:00:05.998857 27973 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:06.002637 28050 leader_election.cc:304] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:05.932027 30668 raft_consensus.cc:491] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
W20250905 20:00:06.009011 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 90584.4us
I20250905 20:00:05.996274 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:06.010784 28151 raft_consensus.cc:3058] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:05.934631 30637 raft_consensus.cc:491] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
W20250905 20:00:05.999707 30543 consensus_meta.cc:287] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.627s user 0.015s sys 0.002s
I20250905 20:00:05.917848 30530 raft_consensus.cc:513] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:06.030166 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:06.031325 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 122179us
I20250905 20:00:06.032622 28717 leader_election.cc:304] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:05.964555 30650 leader_election.cc:290] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:05.909061 30591 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:06.021154 28049 leader_election.cc:400] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299) with higher term. Message: Invalid argument: T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 4. Current term is 5.
I20250905 20:00:06.028980 28156 raft_consensus.cc:2466] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
I20250905 20:00:06.065199 28049 leader_election.cc:403] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
W20250905 20:00:06.018688 30582 consensus_meta.cc:287] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 1.227s user 0.016s sys 0.000s
I20250905 20:00:06.031442 30628 leader_election.cc:290] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:06.011981 30668 raft_consensus.cc:3058] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:06.063753 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:05.988878 30540 raft_consensus.cc:695] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:06.068996 28079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1520) took 2097 ms. Trace:
I20250905 20:00:06.069561 28079 rpcz_store.cc:276] 0905 20:00:03.932719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:03.932933 (+ 214us) service_pool.cc:225] Handling call
0905 20:00:06.030163 (+2097230us) spinlock_profiling.cc:243] Waited 9.89 ms on lock 0x7b54003a9b54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:06.030300 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22750976}
I20250905 20:00:06.010847 30517 raft_consensus.cc:2802] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader election won for term 7
I20250905 20:00:06.033823 30602 raft_consensus.cc:3058] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:06.073974 28745 consensus_queue.cc:260] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:06.069835 30596 raft_consensus.cc:491] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:05.928239 30539 consensus_queue.cc:237] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:06.087689 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 87791.1us
I20250905 20:00:05.995800 30529 consensus_queue.cc:237] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 20:00:06.033715 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:06.015283 30652 leader_election.cc:290] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:06.099073 28151 raft_consensus.cc:2466] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
W20250905 20:00:06.093053 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:06.098599 28752 raft_consensus.cc:2466] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 8.
W20250905 20:00:06.105688 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:06.095314 30685 raft_consensus.cc:2802] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader election won for term 5
I20250905 20:00:05.861667 28077 rpcz_store.cc:276] 0905 20:00:03.719310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:03.722964 (+ 3654us) service_pool.cc:225] Handling call
0905 20:00:03.819738 (+ 96774us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:03.991014 (+171276us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:03.999898 (+ 8884us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:04.346270 (+346372us) log.cc:844] Serialized 26 byte log entry
0905 20:00:05.110156 (+763886us) raft_consensus.cc:1643] Marking committed up to 6
0905 20:00:05.343863 (+233707us) log.cc:844] Serialized 14 byte log entry
0905 20:00:05.358878 (+ 15015us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:05.368331 (+ 9453us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:05.739055 (+370724us) raft_consensus.cc:1683] finished
0905 20:00:05.739080 (+ 25us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:05.740757 (+ 1677us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":177908,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":181248,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":29292,"thread_start_us":977999,"threads_started":2,"wal-append.queue_time_us":849812}
I20250905 20:00:06.086880 30596 raft_consensus.cc:513] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:06.108250 30677 log.cc:927] Time spent T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
W20250905 20:00:06.108445 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 103092us
I20250905 20:00:06.033437 30679 raft_consensus.cc:2802] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250905 20:00:06.112365 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 102672us
I20250905 20:00:06.011281 30594 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:06.089358 30628 raft_consensus.cc:2802] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250905 20:00:06.114446 28151 raft_consensus.cc:1238] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 7. Current term is 8. Ops: []
W20250905 20:00:06.107475 30673 log.cc:927] Time spent T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.132s user 0.000s sys 0.000s
I20250905 20:00:06.032979 30637 raft_consensus.cc:513] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:06.107715 30530 leader_election.cc:290] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:06.065079 30543 raft_consensus.cc:513] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:06.097241 30517 raft_consensus.cc:695] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
W20250905 20:00:06.144466 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:06.151077 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:06.124385 28153 raft_consensus.cc:1273] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250905 20:00:06.153026 28155 raft_consensus.cc:2466] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:06.100544 30672 raft_consensus.cc:2747] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299) with higher term. Message: Invalid argument: T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 4. Current term is 5.
I20250905 20:00:06.157550 28154 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:36164 (request call id 19) took 1471 ms. Trace:
I20250905 20:00:06.114548 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:06.171397 28154 rpcz_store.cc:276] 0905 20:00:04.679394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:04.736577 (+ 57183us) service_pool.cc:225] Handling call
0905 20:00:06.123038 (+1386461us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b540015fed4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc13813 00007fe4a2b801ca 00007fe49c2131c4 00007fe49c213152 00007fe49c213081 00007fe49c212ffc 00007fe49c2122b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:06.150540 (+ 27502us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":296896256}
I20250905 20:00:06.175777 28747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40904 (request call id 630) took 1422 ms. Trace:
I20250905 20:00:06.176265 28747 rpcz_store.cc:276] 0905 20:00:04.753387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:04.800777 (+ 47390us) service_pool.cc:225] Handling call
0905 20:00:04.813626 (+ 12849us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:04.856533 (+ 42907us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:04.856576 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:04.856635 (+ 59us) raft_consensus.cc:1643] Marking committed up to 7
0905 20:00:04.856694 (+ 59us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:06.175521 (+1318827us) spinlock_profiling.cc:243] Waited 42.6 ms on lock 0x7b540048c6d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:06.175573 (+ 52us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:06.175715 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":98049792}
I20250905 20:00:06.124035 30679 raft_consensus.cc:491] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250905 20:00:06.155287 30669 log.cc:927] Time spent T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.322s user 0.001s sys 0.000s
I20250905 20:00:06.152400 30374 leader_election.cc:290] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:06.156551 30628 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250905 20:00:06.198237 28122 connection.cc:232] RPC call timeout handler was delayed by -1.1161436330000924s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21789999999999998s for a response
W20250905 20:00:06.207086 28122 connection.cc:232] RPC call timeout handler was delayed by -1.1091160800000353s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16610000000000003s for a response
I20250905 20:00:06.208353 28051 leader_election.cc:304] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:06.205423 28154 raft_consensus.cc:1273] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250905 20:00:06.166958 30678 raft_consensus.cc:491] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:06.230078 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:06.164019 30680 raft_consensus.cc:2802] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 10 FOLLOWER]: Leader election won for term 10
I20250905 20:00:06.218956 28154 raft_consensus.cc:1273] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250905 20:00:06.234797 30591 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:06.235453 30591 raft_consensus.cc:2802] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:06.235822 30591 raft_consensus.cc:491] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:06.236171 30591 raft_consensus.cc:3058] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Advancing to term 6
W20250905 20:00:06.237664 30557 consensus_meta.cc:287] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.606s user 0.010s sys 0.000s
I20250905 20:00:06.241503 30374 raft_consensus.cc:491] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:06.242141 30374 raft_consensus.cc:513] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:06.196117 30628 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Advancing to term 9
I20250905 20:00:06.249341 28748 raft_consensus.cc:2466] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 20:00:06.100911 30667 consensus_queue.cc:260] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:06.251341 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 218763us
I20250905 20:00:06.189142 30679 raft_consensus.cc:3058] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:06.254055 28717 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:06.195798 30588 leader_election.cc:290] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:06.258117 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 259347us
W20250905 20:00:06.233376 30677 log.cc:927] Time spent T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.056s user 0.000s sys 0.001s
I20250905 20:00:06.260371 28154 raft_consensus.cc:1238] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: []
I20250905 20:00:06.261449 27973 leader_election.cc:304] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:06.269235 28157 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
from {username='slave'} at 127.0.0.1:36164
I20250905 20:00:06.269881 28157 raft_consensus.cc:604] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Received request to transfer leadership to TS e4413ae5e0b64b52a6f360ac580239b9
W20250905 20:00:06.256043 30552 consensus_meta.cc:287] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.110s user 0.016s sys 0.006s
I20250905 20:00:06.256562 30442 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:06.267202 30676 log.cc:927] Time spent T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.196s user 0.001s sys 0.000s
I20250905 20:00:06.243842 30678 raft_consensus.cc:513] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:06.276218 30442 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:06.277340 30442 raft_consensus.cc:2802] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 20:00:06.277684 30678 leader_election.cc:290] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:06.172662 30540 consensus_queue.cc:237] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:06.262913 30685 raft_consensus.cc:695] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
W20250905 20:00:06.283530 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 117634us
I20250905 20:00:06.190547 30689 consensus_queue.cc:1035] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.074s
I20250905 20:00:06.286037 30685 consensus_queue.cc:237] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:06.277769 30442 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:06.287628 28051 leader_election.cc:304] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:06.243356 30661 consensus_queue.cc:1035] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.038s
I20250905 20:00:06.295922 27973 leader_election.cc:304] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:06.267498 30635 raft_consensus.cc:2747] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250905 20:00:06.268792 30680 raft_consensus.cc:695] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 10 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
W20250905 20:00:06.309912 30651 consensus_meta.cc:287] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.585s user 0.008s sys 0.005s
I20250905 20:00:06.324940 27973 leader_election.cc:304] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:06.320401 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:06.267449 30663 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:06.288369 30530 raft_consensus.cc:2802] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election won for term 6
W20250905 20:00:06.356226 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:06.308893 30442 raft_consensus.cc:3058] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:06.252949 30517 consensus_queue.cc:237] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:06.324488 30690 raft_consensus.cc:2802] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election won for term 6
I20250905 20:00:06.398905 30637 leader_election.cc:290] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:06.284374 30552 raft_consensus.cc:513] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:06.410552 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 85581.4us
I20250905 20:00:06.453692 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:06.454501 28008 raft_consensus.cc:3058] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:06.461334 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:06.462193 28010 raft_consensus.cc:3058] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:06.497048 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:06.499456 28002 raft_consensus.cc:1273] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250905 20:00:06.498315 28006 raft_consensus.cc:2391] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:06.501929 28152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:33742 (request call id 684) took 1963 ms. Trace:
W20250905 20:00:06.498239 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 202451us
I20250905 20:00:06.502475 28152 rpcz_store.cc:276] 0905 20:00:04.530393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:04.530636 (+ 243us) service_pool.cc:225] Handling call
0905 20:00:06.493812 (+1963176us) spinlock_profiling.cc:243] Waited 396 ms on lock 0x7b5400316354. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1f6e5 00007fe49dc179ba 00007fe4a2b7e089 00007fe49c207644 00007fe49c2075d2 00007fe49c207501 00007fe49c20747c 00007fe49c206732 00007fe49c018ddc 00007fe49c018216
0905 20:00:06.494109 (+ 297us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":911556864}
W20250905 20:00:06.503629 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 248151us
I20250905 20:00:06.504949 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:06.506668 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:06.507928 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:06.509438 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:06.518898 28010 raft_consensus.cc:2466] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:06.450086 30372 raft_consensus.cc:2802] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election won for term 5
I20250905 20:00:06.444830 30635 raft_consensus.cc:2802] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 20:00:06.402010 30675 consensus_queue.cc:1035] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.107s
I20250905 20:00:06.508574 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:06.544050 28005 raft_consensus.cc:1238] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:06.524205 28003 raft_consensus.cc:1238] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 3. Current term is 4. Ops: []
W20250905 20:00:06.524446 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 67719.4us
I20250905 20:00:06.525231 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:06.545184 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 257597us
I20250905 20:00:06.546720 28001 raft_consensus.cc:2466] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 20:00:06.528402 28158 raft_consensus.cc:2366] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 3. Current term is 4.
I20250905 20:00:06.445120 30530 raft_consensus.cc:695] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:06.372216 30651 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:06.547333 28051 leader_election.cc:304] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 20:00:06.535874 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 203703us
I20250905 20:00:06.520741 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:06.449613 30663 raft_consensus.cc:491] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:06.550468 28009 raft_consensus.cc:3058] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:06.449819 30610 consensus_queue.cc:1046] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.084s
W20250905 20:00:06.585875 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:06.534566 30374 leader_election.cc:290] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:06.585712 28152 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "6db33851e9914ca58560e8902701dca3"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
from {username='slave'} at 127.0.0.1:36164
I20250905 20:00:06.587850 28750 raft_consensus.cc:1273] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250905 20:00:06.604741 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:06.610561 28150 raft_consensus.cc:2391] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 9: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:06.612048 28152 raft_consensus.cc:604] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Received request to transfer leadership to TS e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:06.551085 30459 raft_consensus.cc:3053] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 LEADER]: Stepping down as leader of term 6
I20250905 20:00:06.612597 28152 raft_consensus.cc:612] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250905 20:00:06.545184 30635 raft_consensus.cc:491] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:06.620652 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:06.621752 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 246633us
I20250905 20:00:06.622135 28006 raft_consensus.cc:2391] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
W20250905 20:00:06.622578 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:06.624361 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:06.618558 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:06.553931 30372 raft_consensus.cc:695] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:06.639125 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:06.640421 28007 raft_consensus.cc:2466] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 20:00:06.588553 30618 raft_consensus.cc:491] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:06.545285 30693 consensus_queue.cc:1046] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.098s
W20250905 20:00:06.549562 27894 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet fc7dc431504743d79dbdc8eea83f9dcb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 26b2f874b8e54f08b01a9ed00d5cce05 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
W20250905 20:00:06.650739 27894 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 104721us
I20250905 20:00:06.643057 28004 raft_consensus.cc:1238] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 3. Current term is 4. Ops: []
I20250905 20:00:06.632709 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:06.552815 30663 raft_consensus.cc:513] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:06.623991 30594 raft_consensus.cc:3053] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Stepping down as leader of term 4
I20250905 20:00:06.664448 28007 raft_consensus.cc:1238] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 LEADER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250905 20:00:06.624573 30635 raft_consensus.cc:3058] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:06.619864 30459 raft_consensus.cc:738] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:06.676632 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:06.678921 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 132918us
W20250905 20:00:06.685635 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 149510us
I20250905 20:00:06.643244 30678 raft_consensus.cc:2747] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:06.692853 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:06.693921 28151 raft_consensus.cc:2466] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 8.
I20250905 20:00:06.545720 30680 consensus_queue.cc:237] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:06.698908 28009 raft_consensus.cc:2466] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 20:00:06.700484 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:06.701136 28151 raft_consensus.cc:3058] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:06.696502 28718 leader_election.cc:304] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:06.705360 27974 leader_election.cc:304] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 20:00:06.707207 30691 log.cc:927] Time spent T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.298s user 0.001s sys 0.000s
I20250905 20:00:06.707600 30696 raft_consensus.cc:491] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:06.709384 28717 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:06.678639 30594 raft_consensus.cc:738] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:06.718302 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:06.721474 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:06.722301 28009 raft_consensus.cc:2466] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
W20250905 20:00:06.713220 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 166331us
I20250905 20:00:06.724406 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:06.725306 28007 raft_consensus.cc:2466] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
W20250905 20:00:06.726293 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 162220us
I20250905 20:00:06.662968 30618 raft_consensus.cc:513] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:06.736646 28156 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33752 (request call id 1594) took 1199 ms. Trace:
I20250905 20:00:06.737129 28156 rpcz_store.cc:276] 0905 20:00:05.537010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:05.551077 (+ 14067us) service_pool.cc:225] Handling call
0905 20:00:06.736447 (+1185370us) spinlock_profiling.cc:243] Waited 330 ms on lock 0x7b54002d0854. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:06.736583 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":759121280}
I20250905 20:00:06.740646 28081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1533) took 1677 ms. Trace:
I20250905 20:00:06.741305 28081 rpcz_store.cc:276] 0905 20:00:05.062998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:05.063231 (+ 233us) service_pool.cc:225] Handling call
0905 20:00:05.063733 (+ 502us) raft_consensus.cc:1469] Updating replica for 2 ops
0905 20:00:05.277734 (+214001us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:05.277768 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0905 20:00:05.278531 (+ 763us) log.cc:844] Serialized 52 byte log entry
0905 20:00:05.600262 (+321731us) raft_consensus.cc:1643] Marking committed up to 5
0905 20:00:05.644776 (+ 44514us) log.cc:844] Serialized 14 byte log entry
0905 20:00:05.653307 (+ 8531us) log.cc:844] Serialized 14 byte log entry
0905 20:00:05.823582 (+170275us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:06.740051 (+916469us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b54001270d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:06.740129 (+ 78us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:06.740300 (+ 171us) raft_consensus.cc:1683] finished
0905 20:00:06.740321 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:06.740556 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17118,"prepare.queue_time_us":49015,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":2648,"raft.queue_time_us":201508,"raft.run_cpu_time_us":340,"raft.run_wall_time_us":25032,"spinlock_wait_cycles":319370240,"thread_start_us":397449,"threads_started":2,"wal-append.queue_time_us":316371}
W20250905 20:00:06.837199 28715 leader_election.cc:336] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 2.142s (SENT)
I20250905 20:00:06.873471 28151 raft_consensus.cc:2466] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 7.
I20250905 20:00:06.873474 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:06.875569 28717 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:06.723702 30695 raft_consensus.cc:2802] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 20:00:06.885183 28156 raft_consensus.cc:1238] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: []
W20250905 20:00:06.893891 28715 leader_election.cc:336] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.700s (SENT)
I20250905 20:00:06.894394 28715 leader_election.cc:304] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:06.677915 30530 consensus_queue.cc:237] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:06.875800 30679 consensus_meta.cc:287] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.615s user 0.026s sys 0.002s
I20250905 20:00:06.729630 30710 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:06.735786 30552 leader_election.cc:290] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:06.903935 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:06.878367 30697 log.cc:927] Time spent T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.433s user 0.000s sys 0.001s
I20250905 20:00:06.908492 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:06.721031 30696 raft_consensus.cc:513] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:06.727664 30372 consensus_queue.cc:237] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:06.910080 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 186272us
W20250905 20:00:06.916549 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 220157us
I20250905 20:00:07.098017 28747 raft_consensus.cc:1273] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20250905 20:00:07.171747 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:07.172614 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:06.984393 28004 raft_consensus.cc:1238] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:07.123164 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:07.027362 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:07.171367 28051 leader_election.cc:304] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:07.174785 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 20:00:07.175480 28002 consensus_meta.cc:287] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.661s user 0.012s sys 0.000s
I20250905 20:00:06.938939 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:07.020269 30675 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:07.021304 28125 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:07.177984 30419 raft_consensus.cc:2747] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:07.178400 30688 consensus_queue.cc:1035] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250905 20:00:07.178674 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:07.178982 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:07.179607 28080 raft_consensus.cc:2466] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 20:00:07.177731 30675 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:07.180588 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:07.180744 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:07.181188 28083 raft_consensus.cc:2466] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 20:00:07.181917 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:07.182256 30675 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:07.181988 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:07.182600 28081 raft_consensus.cc:2466] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 20:00:07.182730 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:07.182960 28083 raft_consensus.cc:2466] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:07.183318 28084 raft_consensus.cc:3058] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
W20250905 20:00:07.178628 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 157419us
I20250905 20:00:07.184109 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:07.184566 28125 leader_election.cc:304] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:07.185009 28077 raft_consensus.cc:2408] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250905 20:00:06.906677 30678 consensus_queue.cc:1046] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.285s
I20250905 20:00:07.185354 30419 raft_consensus.cc:2747] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:06.892083 30459 consensus_queue.cc:260] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:07.172106 28006 raft_consensus.cc:2466] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
W20250905 20:00:07.080775 28008 consensus_meta.cc:287] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.587s user 0.010s sys 0.006s
I20250905 20:00:07.187249 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:07.187544 28008 raft_consensus.cc:2466] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 20:00:07.187878 28078 raft_consensus.cc:2466] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:06.941005 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:07.176237 28002 raft_consensus.cc:2466] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 8.
I20250905 20:00:07.188966 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:07.189131 28051 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:07.189551 28085 raft_consensus.cc:3058] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 5
I20250905 20:00:07.186703 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:07.193001 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:07.194029 28082 raft_consensus.cc:2466] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 3.
I20250905 20:00:07.195370 28125 leader_election.cc:304] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:06.897087 30693 raft_consensus.cc:3053] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 LEADER]: Stepping down as leader of term 5
I20250905 20:00:07.197789 30419 consensus_queue.cc:1046] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 20:00:06.910004 30695 raft_consensus.cc:695] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:07.025516 30612 consensus_queue.cc:1035] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.173s
I20250905 20:00:06.902534 30507 raft_consensus.cc:491] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:07.199353 30552 consensus_queue.cc:1046] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 20:00:07.192831 30679 raft_consensus.cc:513] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:07.200503 30442 consensus_meta.cc:287] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.791s user 0.012s sys 0.000s
I20250905 20:00:06.980916 30717 raft_consensus.cc:2802] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 20:00:07.209699 28051 leader_election.cc:304] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 20:00:07.211634 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 317730us
I20250905 20:00:07.186862 30459 raft_consensus.cc:3058] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Advancing to term 7
W20250905 20:00:07.081051 30706 log.cc:927] Time spent T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.169s user 0.001s sys 0.000s
W20250905 20:00:07.096913 30714 log.cc:927] Time spent T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.111s user 0.001s sys 0.000s
W20250905 20:00:07.172995 30591 consensus_meta.cc:287] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.936s user 0.013s sys 0.001s
W20250905 20:00:07.111052 30628 consensus_meta.cc:287] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.848s user 0.009s sys 0.002s
W20250905 20:00:07.082633 30699 log.cc:927] Time spent T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.384s user 0.002s sys 0.000s
W20250905 20:00:06.994396 30559 consensus_meta.cc:287] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 1.072s user 0.008s sys 0.006s
I20250905 20:00:07.034374 30577 raft_consensus.cc:2802] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader election won for term 7
I20250905 20:00:07.283479 28084 raft_consensus.cc:2466] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:06.992697 30619 raft_consensus.cc:3053] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 20:00:07.030650 30663 leader_election.cc:290] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:07.087038 30721 raft_consensus.cc:2747] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250905 20:00:07.238624 28715 leader_election.cc:336] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.816s (SENT)
I20250905 20:00:07.287812 28715 leader_election.cc:304] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:07.026760 30709 raft_consensus.cc:2802] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250905 20:00:07.215056 30675 raft_consensus.cc:2747] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250905 20:00:07.212601 28085 raft_consensus.cc:2466] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:07.293696 28085 raft_consensus.cc:1273] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20250905 20:00:07.207732 30240 consensus_meta.cc:287] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.317s user 0.009s sys 0.005s
I20250905 20:00:07.241008 28051 leader_election.cc:304] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:07.092940 30542 raft_consensus.cc:491] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:07.192817 30678 raft_consensus.cc:2747] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250905 20:00:07.246861 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:07.210606 30693 raft_consensus.cc:738] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:07.236514 30507 raft_consensus.cc:513] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:06.910540 30594 consensus_queue.cc:260] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 20:00:07.213624 30710 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:07.247119 30442 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:07.247360 30635 consensus_meta.cc:287] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.549s user 0.008s sys 0.004s
W20250905 20:00:07.394290 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 153393us
I20250905 20:00:07.243607 30717 raft_consensus.cc:491] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:07.267805 30695 consensus_queue.cc:237] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:07.270982 30679 leader_election.cc:290] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:07.222098 30718 raft_consensus.cc:491] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:07.274116 30543 leader_election.cc:290] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:07.276880 30668 consensus_meta.cc:287] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.181s user 0.013s sys 0.004s
W20250905 20:00:07.277073 30686 log.cc:927] Time spent T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 1.159s user 0.003s sys 0.015s
I20250905 20:00:07.277709 30374 raft_consensus.cc:2747] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:07.254498 30591 raft_consensus.cc:513] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:07.285120 30559 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:07.396250 28051 leader_election.cc:304] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 20:00:07.396656 28751 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.691s user 0.002s sys 0.015s
W20250905 20:00:07.397739 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 74081.7us
W20250905 20:00:07.435184 28751 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.730s user 0.003s sys 0.015s
I20250905 20:00:07.317541 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:07.436573 28051 leader_election.cc:304] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:07.295189 30709 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:07.286787 30619 raft_consensus.cc:738] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:07.408527 28085 raft_consensus.cc:1273] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250905 20:00:07.286787 30725 raft_consensus.cc:2747] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:07.411428 27973 leader_election.cc:304] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:07.256107 30628 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:07.419513 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:07.369778 28746 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.969s user 0.006s sys 0.010s
W20250905 20:00:07.452934 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 257646us
I20250905 20:00:07.317478 30542 raft_consensus.cc:513] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:07.458194 28748 raft_consensus.cc:1238] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 5. Current term is 6. Ops: []
W20250905 20:00:07.440439 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:07.406847 30675 raft_consensus.cc:2802] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 20:00:07.401983 30717 raft_consensus.cc:3058] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:07.281423 28153 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2129) took 1525 ms. Trace:
I20250905 20:00:07.401858 30635 raft_consensus.cc:513] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:07.310058 30240 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:07.445400 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:07.455384 28125 leader_election.cc:304] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:07.431071 30678 raft_consensus.cc:491] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:07.407217 30713 consensus_queue.cc:1046] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20250905 20:00:07.454288 27973 leader_election.cc:304] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:07.493266 27973 leader_election.cc:304] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:07.494128 27973 leader_election.cc:304] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:07.495121 27973 leader_election.cc:304] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:07.443101 30709 raft_consensus.cc:3058] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Advancing to term 9
W20250905 20:00:07.457947 28746 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.057s user 0.007s sys 0.011s
I20250905 20:00:07.443535 30679 raft_consensus.cc:2747] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 20:00:07.490697 28125 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:07.429757 30668 raft_consensus.cc:513] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:07.481941 30675 raft_consensus.cc:491] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:07.483362 28153 rpcz_store.cc:276] 0905 20:00:05.756054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:06.236381 (+480327us) service_pool.cc:225] Handling call
0905 20:00:06.236683 (+ 302us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:06.237032 (+ 349us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:06.237075 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:06.237588 (+ 513us) log.cc:844] Serialized 26 byte log entry
0905 20:00:06.358491 (+120903us) raft_consensus.cc:1643] Marking committed up to 2
0905 20:00:06.358558 (+ 67us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:06.358662 (+ 104us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:07.281205 (+922543us) raft_consensus.cc:1683] finished
0905 20:00:07.281226 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:07.281360 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1177,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"thread_start_us":119766,"threads_started":1,"wal-append.queue_time_us":380339}
I20250905 20:00:07.484359 30577 raft_consensus.cc:695] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 7 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:07.498268 27973 leader_election.cc:304] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:07.443363 30374 raft_consensus.cc:491] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:07.480482 30419 consensus_queue.cc:1230] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer cd764bfeebac4cc9ad851cd4a6a7d3b0 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250905 20:00:07.492233 28076 raft_consensus.cc:2391] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 8: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:07.513617 28125 leader_election.cc:304] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:07.514621 28079 raft_consensus.cc:2391] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 4: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:07.506908 30530 raft_consensus.cc:2802] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250905 20:00:07.520561 28122 leader_election.cc:336] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 2.466s (SENT)
I20250905 20:00:07.521147 30507 leader_election.cc:290] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:07.416039 30718 raft_consensus.cc:513] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:07.456310 30693 consensus_queue.cc:260] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:07.524587 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 285958us
W20250905 20:00:08.095733 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 659171us
I20250905 20:00:07.497725 30540 raft_consensus.cc:2802] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 20:00:08.096735 28051 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 20:00:08.108554 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 610276us
I20250905 20:00:08.109530 27973 leader_election.cc:304] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 20:00:07.765260 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:08.103345 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:08.111889 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:08.112690 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:08.103528 28122 leader_election.cc:304] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:07.669049 28155 raft_consensus.cc:1238] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 4. Current term is 6. Ops: []
W20250905 20:00:08.104116 28050 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.668s user 0.016s sys 0.019s
W20250905 20:00:08.113583 28050 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.678s user 0.016s sys 0.020s
W20250905 20:00:08.101315 28718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.581s user 0.002s sys 0.009s
W20250905 20:00:08.099320 27975 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.127s user 0.058s sys 0.071s
W20250905 20:00:08.113890 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:08.114614 28718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.594s user 0.002s sys 0.010s
W20250905 20:00:08.113893 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 678100us
W20250905 20:00:08.114833 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:08.114995 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:07.503173 30470 raft_consensus.cc:2802] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250905 20:00:08.041250 28110 heartbeater.cc:646] Failed to heartbeat to 127.26.58.190:38969 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.26.58.190:38969 timed out after 15.000s (SENT)
W20250905 20:00:08.084936 30724 log.cc:927] Time spent T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.664s user 0.001s sys 0.001s
W20250905 20:00:08.129429 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 615812us
I20250905 20:00:08.129834 27973 leader_election.cc:304] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:08.130275 28125 leader_election.cc:304] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 20:00:08.131088 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 610467us
I20250905 20:00:07.513172 30588 raft_consensus.cc:2762] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20250905 20:00:08.119184 28154 raft_consensus.cc:1238] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: []
I20250905 20:00:08.146762 30693 raft_consensus.cc:3058] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:07.863960 30542 leader_election.cc:290] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:07.902419 30577 consensus_queue.cc:237] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:08.123668 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:08.124369 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:08.124558 30718 leader_election.cc:290] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 20:00:08.104000 30680 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.011s user 0.024s sys 0.000s
W20250905 20:00:08.154078 30680 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.062s user 0.024s sys 0.000s
I20250905 20:00:07.501508 30678 raft_consensus.cc:513] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:07.903184 30720 log.cc:927] Time spent T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.822s user 0.011s sys 0.004s
W20250905 20:00:08.158457 27975 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.186s user 0.058s sys 0.072s
W20250905 20:00:08.158840 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 2.19118e+06us
I20250905 20:00:08.126736 30470 raft_consensus.cc:491] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:07.896338 30442 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:07.705015 30696 leader_election.cc:290] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:07.703658 30726 raft_consensus.cc:2747] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250905 20:00:08.133836 30517 raft_consensus.cc:2802] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:08.078926 30668 leader_election.cc:290] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:07.522147 30628 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 9 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:08.138856 30741 raft_consensus.cc:491] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:08.140066 30688 raft_consensus.cc:2802] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election won for term 5
I20250905 20:00:08.140848 30739 raft_consensus.cc:2747] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:08.140879 30738 raft_consensus.cc:2747] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:08.141564 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:07.868538 28746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40918 (request call id 560) took 1953 ms. Trace:
I20250905 20:00:07.865307 30240 leader_election.cc:290] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:07.527127 30660 raft_consensus.cc:3053] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 20:00:07.637753 30619 consensus_queue.cc:260] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:08.094689 30559 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:07.695815 30635 leader_election.cc:290] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:08.106541 30540 raft_consensus.cc:491] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:08.150722 28151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2137) took 1267 ms. Trace:
I20250905 20:00:08.106806 30530 raft_consensus.cc:491] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:08.253201 30726 raft_consensus.cc:491] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:08.253386 30530 raft_consensus.cc:3058] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
W20250905 20:00:08.256345 30459 consensus_meta.cc:287] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.003s user 0.009s sys 0.001s
I20250905 20:00:08.257009 30695 raft_consensus.cc:2802] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election won for term 5
W20250905 20:00:08.258074 30709 consensus_meta.cc:287] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.733s user 0.007s sys 0.003s
I20250905 20:00:08.258502 30540 raft_consensus.cc:3058] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
W20250905 20:00:08.120642 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:08.260437 30667 raft_consensus.cc:3058] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:08.263159 28082 raft_consensus.cc:1273] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250905 20:00:08.264336 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:08.264858 28082 raft_consensus.cc:3058] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:08.242146 30741 raft_consensus.cc:513] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:08.267822 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 20:00:08.268697 28749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40926 (request call id 484) took 2255 ms (client timeout 1945 ms). Trace:
W20250905 20:00:08.257027 28752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40904 (request call id 638) took 2464 ms (client timeout 2133 ms). Trace:
I20250905 20:00:07.513780 30552 raft_consensus.cc:2747] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:08.157231 28125 leader_election.cc:304] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:07.516637 30713 raft_consensus.cc:2747] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:08.162168 28718 leader_election.cc:304] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:08.156533 30732 raft_consensus.cc:2802] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 20:00:08.211427 28751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40904 (request call id 644) took 1507 ms. Trace:
W20250905 20:00:08.209369 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 112726us
I20250905 20:00:08.292266 28051 leader_election.cc:304] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:07.530295 30374 raft_consensus.cc:513] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:08.222393 28152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:36164 (request call id 21) took 1670 ms. Trace:
I20250905 20:00:08.290254 28751 rpcz_store.cc:276] 0905 20:00:06.704303 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:06.704427 (+ 124us) service_pool.cc:225] Handling call
0905 20:00:06.704763 (+ 336us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:07.435579 (+730816us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:07.435610 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:07.436147 (+ 537us) log.cc:844] Serialized 26 byte log entry
0905 20:00:07.694121 (+257974us) raft_consensus.cc:1643] Marking committed up to 6
0905 20:00:07.694176 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:07.694267 (+ 91us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:08.211185 (+516918us) raft_consensus.cc:1683] finished
0905 20:00:08.211207 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:08.211368 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":315819,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":54909,"thread_start_us":937832,"threads_started":2,"wal-append.queue_time_us":755558}
I20250905 20:00:08.294610 28152 rpcz_store.cc:276] 0905 20:00:06.551856 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:06.552223 (+ 367us) service_pool.cc:225] Handling call
0905 20:00:08.222160 (+1669937us) spinlock_profiling.cc:243] Waited 25.1 ms on lock 0x7b5400200654. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc13813 00007fe4a2b801ca 00007fe49c2131c4 00007fe49c213152 00007fe49c213081 00007fe49c212ffc 00007fe49c2122b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:08.222314 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":57648896}
I20250905 20:00:08.224225 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:08.296154 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:08.208177 30745 raft_consensus.cc:491] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:08.195370 30470 raft_consensus.cc:3058] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:08.253885 30726 raft_consensus.cc:513] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:08.217358 30542 raft_consensus.cc:491] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:08.250294 28151 rpcz_store.cc:276] 0905 20:00:06.883802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:06.900983 (+ 17181us) service_pool.cc:225] Handling call
0905 20:00:06.901479 (+ 496us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:06.930887 (+ 29408us) raft_consensus.cc:1532] Early marking committed up to index 3
0905 20:00:07.308130 (+377243us) log.cc:844] Serialized 14 byte log entry
0905 20:00:08.100172 (+792042us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:08.117221 (+ 17049us) log.cc:844] Serialized 26 byte log entry
0905 20:00:08.117435 (+ 214us) raft_consensus.cc:1643] Marking committed up to 3
0905 20:00:08.117491 (+ 56us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:08.117598 (+ 107us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:08.150464 (+ 32866us) raft_consensus.cc:1683] finished
0905 20:00:08.150493 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:08.150651 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":225661,"prepare.queue_time_us":99369,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":58,"raft.queue_time_us":501798,"raft.run_cpu_time_us":326,"raft.run_wall_time_us":22645,"thread_start_us":516564,"threads_started":2,"wal-append.queue_time_us":725141}
I20250905 20:00:08.194972 30736 raft_consensus.cc:2802] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 20:00:07.517163 30675 raft_consensus.cc:3058] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
W20250905 20:00:08.269085 28749 rpcz_store.cc:269] 0905 20:00:06.013099 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:06.013292 (+ 193us) service_pool.cc:225] Handling call
0905 20:00:08.267657 (+2254365us) spinlock_profiling.cc:243] Waited 3.14 ms on lock 0x7b5800304598. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b7db60 00007fe4a2b7d444 00007fe49c2050c4 00007fe49c205052
0905 20:00:08.268620 (+ 963us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7224576}
I20250905 20:00:08.239162 30734 raft_consensus.cc:2747] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:08.269207 30743 consensus_queue.cc:1046] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20250905 20:00:08.219349 30718 consensus_queue.cc:1046] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
W20250905 20:00:08.270017 28752 rpcz_store.cc:269] 0905 20:00:05.792982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:05.793388 (+ 406us) service_pool.cc:225] Handling call
0905 20:00:08.256815 (+2463427us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x7b54004321d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:08.256943 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":460901248}
W20250905 20:00:08.323944 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 194168us
I20250905 20:00:08.269260 28085 raft_consensus.cc:1273] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20250905 20:00:08.262256 30717 consensus_meta.cc:287] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.770s user 0.007s sys 0.001s
I20250905 20:00:08.249186 30660 raft_consensus.cc:738] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:08.236508 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:08.297771 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
W20250905 20:00:08.299768 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 137662us
I20250905 20:00:08.336957 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:08.309283 30552 raft_consensus.cc:2802] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 20:00:08.250703 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:08.303133 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 187756us
W20250905 20:00:08.304149 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 146985us
I20250905 20:00:08.288558 30695 raft_consensus.cc:695] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:08.344516 28154 raft_consensus.cc:1273] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250905 20:00:08.224987 30710 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:08.307379 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:08.312255 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:08.288818 30678 leader_election.cc:290] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:08.315622 28051 leader_election.cc:400] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Vote denied by peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421) with higher term. Message: Invalid argument: T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 6. Current term is 7.
I20250905 20:00:08.348129 28051 leader_election.cc:403] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20250905 20:00:08.318332 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:08.349612 28125 leader_election.cc:304] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:08.321528 30459 raft_consensus.cc:491] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:08.298553 30732 raft_consensus.cc:491] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:08.301245 30602 raft_consensus.cc:2747] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:08.323385 28082 raft_consensus.cc:2466] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 20:00:08.306970 30713 raft_consensus.cc:2747] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:08.246421 30688 raft_consensus.cc:695] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:08.258544 30709 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:08.309389 30736 raft_consensus.cc:491] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:08.346664 28151 raft_consensus.cc:2391] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 4: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:08.341096 30718 consensus_queue.cc:1230] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Peer cd764bfeebac4cc9ad851cd4a6a7d3b0 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20250905 20:00:08.353478 28051 leader_election.cc:304] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:08.354063 28156 raft_consensus.cc:1238] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 4. Current term is 5. Ops: []
W20250905 20:00:08.320452 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 194430us
I20250905 20:00:08.311401 30542 raft_consensus.cc:513] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:08.340281 30552 raft_consensus.cc:491] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:08.343621 30741 leader_election.cc:290] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:08.353305 30734 raft_consensus.cc:2747] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421) with higher term. Message: Invalid argument: T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 6. Current term is 7.
W20250905 20:00:08.380609 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 118431us
I20250905 20:00:08.352625 30537 consensus_queue.cc:1046] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
W20250905 20:00:08.400394 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 46992.7us
W20250905 20:00:08.406059 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 79672.5us
I20250905 20:00:08.305687 30745 raft_consensus.cc:513] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:08.364737 30732 raft_consensus.cc:3058] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:08.403359 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
W20250905 20:00:08.379802 28122 leader_election.cc:336] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 2.138s (SENT)
I20250905 20:00:08.412868 28122 leader_election.cc:304] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
W20250905 20:00:08.416110 27975 connection.cc:232] RPC call timeout handler was delayed by -1.8483791730000121s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2009s for a response
I20250905 20:00:08.416420 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 20:00:08.416813 27975 connection.cc:232] RPC call timeout handler was delayed by -1.704262184000072s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1552s for a response
W20250905 20:00:08.417261 27975 connection.cc:232] RPC call timeout handler was delayed by -1.6539791729999251s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22250000000000003s for a response
W20250905 20:00:08.417690 27975 connection.cc:232] RPC call timeout handler was delayed by -1.6443171140000459s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15000000000000002s for a response
W20250905 20:00:08.418130 27975 connection.cc:232] RPC call timeout handler was delayed by -1.5969256289999976s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1507s for a response
W20250905 20:00:08.418607 27975 connection.cc:232] RPC call timeout handler was delayed by -1.4884940580000148s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2066s for a response
I20250905 20:00:08.339730 30717 raft_consensus.cc:513] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:08.419060 27975 connection.cc:232] RPC call timeout handler was delayed by -1.393837334000068s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1779s for a response
I20250905 20:00:08.408730 28051 leader_election.cc:304] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
W20250905 20:00:08.419636 27975 connection.cc:232] RPC call timeout handler was delayed by -1.3524791730000061s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.256s for a response
I20250905 20:00:08.417280 28752 raft_consensus.cc:2466] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
I20250905 20:00:08.392622 30736 raft_consensus.cc:3058] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:08.403508 30678 raft_consensus.cc:2747] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 20:00:08.400019 30602 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
W20250905 20:00:08.420181 27975 connection.cc:232] RPC call timeout handler was delayed by -1.3281373340000755s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18520000000000003s for a response
I20250905 20:00:08.375522 30530 raft_consensus.cc:513] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:08.434175 28051 leader_election.cc:304] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:08.374914 30459 raft_consensus.cc:513] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:08.424809 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 45046.6us
I20250905 20:00:08.405800 30552 raft_consensus.cc:3058] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:08.435091 27974 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 20:00:08.432170 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 106799us
W20250905 20:00:08.436264 27975 connection.cc:232] RPC call timeout handler was delayed by -1.325060709000013s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1912s for a response
I20250905 20:00:08.437659 28049 leader_election.cc:304] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
W20250905 20:00:08.438472 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 88941.1us
I20250905 20:00:08.439127 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:08.405864 30668 raft_consensus.cc:2802] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election won for term 4
W20250905 20:00:08.443688 27975 connection.cc:232] RPC call timeout handler was delayed by -1.2830164690000174s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.163s for a response
W20250905 20:00:08.444206 27975 connection.cc:232] RPC call timeout handler was delayed by -1.2354791729999306s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.269s for a response
I20250905 20:00:08.444151 28125 leader_election.cc:304] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 20:00:08.445094 27975 connection.cc:232] RPC call timeout handler was delayed by -1.2147791730000108s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.27130000000000004s for a response
W20250905 20:00:08.445516 27975 connection.cc:232] RPC call timeout handler was delayed by -1.1970171140001185s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19970000000000002s for a response
W20250905 20:00:08.445916 27975 connection.cc:232] RPC call timeout handler was delayed by -1.0461522979999245s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21330000000000002s for a response
I20250905 20:00:08.448041 28125 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:08.430717 30537 consensus_queue.cc:1230] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer cd764bfeebac4cc9ad851cd4a6a7d3b0 log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20250905 20:00:08.451491 28051 leader_election.cc:304] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:08.461418 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:08.441613 30749 log.cc:927] Time spent T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.080s user 0.001s sys 0.000s
I20250905 20:00:08.464061 28152 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
from {username='slave'} at 127.0.0.1:36164
I20250905 20:00:08.438796 30635 raft_consensus.cc:2802] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:08.415506 30628 consensus_queue.cc:1035] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20250905 20:00:08.446718 30755 raft_consensus.cc:2747] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250905 20:00:08.492424 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 46043.9us
I20250905 20:00:08.452410 30678 raft_consensus.cc:2747] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250905 20:00:08.404281 30754 consensus_queue.cc:1035] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
I20250905 20:00:08.506923 28752 raft_consensus.cc:1238] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 7 LEADER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:08.453985 30758 raft_consensus.cc:2747] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:08.443353 30602 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:08.474718 30537 raft_consensus.cc:2802] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:08.473598 30635 raft_consensus.cc:491] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:08.246994 28746 rpcz_store.cc:276] 0905 20:00:05.914990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:05.965209 (+ 50219us) service_pool.cc:225] Handling call
0905 20:00:06.077595 (+112386us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:06.368724 (+291129us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:06.370880 (+ 2156us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:06.400255 (+ 29375us) log.cc:844] Serialized 26 byte log entry
0905 20:00:07.518060 (+1117805us) raft_consensus.cc:1643] Marking committed up to 7
0905 20:00:07.529692 (+ 11632us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:07.670804 (+141112us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:07.703145 (+ 32341us) raft_consensus.cc:1683] finished
0905 20:00:07.731880 (+ 28735us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:07.845934 (+114054us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1651,"raft.queue_time_us":163820,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":94,"thread_start_us":1318710,"threads_started":2,"wal-append.queue_time_us":1117965}
I20250905 20:00:08.399209 30660 consensus_queue.cc:260] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:08.505448 28743 raft_consensus.cc:2391] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 4: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
W20250905 20:00:08.536134 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:08.544097 28122 leader_election.cc:336] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.515s (SENT)
W20250905 20:00:08.572153 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:08.572255 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 44091.2us
W20250905 20:00:08.572887 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:08.526629 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 78713.1us
I20250905 20:00:08.521417 30374 leader_election.cc:290] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 20:00:08.582906 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:08.589233 30752 raft_consensus.cc:2802] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250905 20:00:08.589792 30752 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:08.590157 30752 raft_consensus.cc:3058] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Advancing to term 9
W20250905 20:00:08.593850 30728 log.cc:927] Time spent T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 1.289s user 0.005s sys 0.005s
I20250905 20:00:08.547662 30635 raft_consensus.cc:3058] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:08.525385 30679 raft_consensus.cc:2747] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:08.546170 30726 leader_election.cc:290] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:08.529078 30537 raft_consensus.cc:491] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:08.599669 28157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2139) took 1710 ms. Trace:
I20250905 20:00:08.463210 30695 consensus_queue.cc:237] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:08.604092 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:08.605485 28158 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33752 (request call id 1583) took 3205 ms (client timeout 2713 ms). Trace:
I20250905 20:00:08.527772 30713 raft_consensus.cc:491] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:08.614329 28158 rpcz_store.cc:269] 0905 20:00:05.400045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:05.401556 (+ 1511us) service_pool.cc:225] Handling call
0905 20:00:08.605270 (+3203714us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b540026cad4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:08.605414 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2467532160}
I20250905 20:00:08.594410 30755 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:08.581344 30764 raft_consensus.cc:2802] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader election won for term 8
W20250905 20:00:08.624404 28122 leader_election.cc:336] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.577s (SENT)
I20250905 20:00:08.451167 30688 consensus_queue.cc:237] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:08.620261 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:08.613821 28157 rpcz_store.cc:276] 0905 20:00:06.889508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:06.889742 (+ 234us) service_pool.cc:225] Handling call
0905 20:00:06.934032 (+ 44290us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:06.955828 (+ 21796us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:06.955873 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:06.980236 (+ 24363us) log.cc:844] Serialized 26 byte log entry
0905 20:00:07.303246 (+323010us) raft_consensus.cc:1643] Marking committed up to 7
0905 20:00:07.303286 (+ 40us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:07.303379 (+ 93us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:08.593087 (+1289708us) spinlock_profiling.cc:243] Waited 19.5 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49d85891e 00007fe49d857c66 00007fe49d857888 00007fe49d8612c1 00007fe49d861279 00007fe49d861209
0905 20:00:08.599392 (+ 6305us) raft_consensus.cc:1683] finished
0905 20:00:08.599413 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:08.599612 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21243,"raft.queue_time_us":48720,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":8057,"spinlock_wait_cycles":44961152,"thread_start_us":278608,"threads_started":1,"wal-append.queue_time_us":302722}
I20250905 20:00:08.610137 30745 leader_election.cc:290] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:08.629051 30753 log.cc:927] Time spent T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.173s user 0.001s sys 0.001s
I20250905 20:00:08.598268 30660 raft_consensus.cc:3058] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:08.618148 30537 raft_consensus.cc:3058] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:08.682984 30651 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:08.634377 30542 leader_election.cc:290] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:08.691200 30530 leader_election.cc:290] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:08.644130 30591 leader_election.cc:290] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:08.694833 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:08.619619 30709 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 9 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:08.701610 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:08.716749 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:08.724902 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 273507us
W20250905 20:00:08.726054 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:08.726333 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:08.727994 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:08.728493 28084 raft_consensus.cc:1238] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20250905 20:00:08.729820 28082 raft_consensus.cc:1238] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 3. Current term is 5. Ops: [3.5-3.5]
I20250905 20:00:08.732955 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:08.639892 30764 raft_consensus.cc:695] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:08.740566 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:08.741498 28084 raft_consensus.cc:2466] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 20:00:08.743000 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:08.743877 28085 raft_consensus.cc:2374] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6. Re-sending same reply.
W20250905 20:00:08.745791 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 223828us
I20250905 20:00:08.748453 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:08.741592 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:08.750770 28082 raft_consensus.cc:2466] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 20:00:08.752640 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:08.767956 28082 raft_consensus.cc:2391] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 5: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:08.725107 30459 leader_election.cc:290] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:08.769711 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:08.732726 30717 leader_election.cc:290] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:08.694187 30736 raft_consensus.cc:513] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:08.626217 30713 raft_consensus.cc:513] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:08.774724 28084 raft_consensus.cc:2466] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 20:00:08.787513 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:08.786543 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:08.789408 28082 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 20:00:08.789592 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:08.790709 28084 raft_consensus.cc:2366] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a for earlier term 9. Current term is 10.
I20250905 20:00:08.635048 30755 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:08.794660 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:08.836637 28085 raft_consensus.cc:2391] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:08.732590 30470 raft_consensus.cc:513] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:08.797961 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:08.839601 28082 raft_consensus.cc:2466] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 20:00:08.839515 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:08.804368 28122 leader_election.cc:336] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.661s (SENT)
I20250905 20:00:08.840283 28084 raft_consensus.cc:2466] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 20:00:08.791329 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 20:00:08.812057 28076 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1542) took 1634 ms (client timeout 1507 ms). Trace:
W20250905 20:00:08.844797 28076 rpcz_store.cc:269] 0905 20:00:07.177933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:07.178373 (+ 440us) service_pool.cc:225] Handling call
0905 20:00:08.811846 (+1633473us) spinlock_profiling.cc:243] Waited 303 ms on lock 0x7b540029b154. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:08.811983 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":697022208}
W20250905 20:00:08.814291 28150 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33752 (request call id 1589) took 3351 ms (client timeout 3105 ms). Trace:
W20250905 20:00:08.846124 28150 rpcz_store.cc:269] 0905 20:00:05.462688 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:05.464160 (+ 1472us) service_pool.cc:225] Handling call
0905 20:00:08.814084 (+3349924us) spinlock_profiling.cc:243] Waited 990 ms on lock 0x7b5400200654. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:08.814206 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2277147520}
I20250905 20:00:08.816295 30764 consensus_queue.cc:237] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:08.821002 28715 leader_election.cc:336] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.948s (SENT)
I20250905 20:00:08.810272 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:08.853255 28080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1548) took 1673 ms. Trace:
I20250905 20:00:08.853704 28080 rpcz_store.cc:276] 0905 20:00:07.179786 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:07.180286 (+ 500us) service_pool.cc:225] Handling call
0905 20:00:08.810137 (+1629851us) spinlock_profiling.cc:243] Waited 2.11 ms on lock 0x7b5800204f98. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac6721e 000000000035a3c9 00007fe4a2b08310 00007fe4a2bceed4 00007fe4a2bcef41 00007fe4a2b7db60 00007fe4a2b7d444 00007fe49c2050c4 00007fe49c205052
0905 20:00:08.853167 (+ 43030us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4875264}
W20250905 20:00:08.805281 30540 consensus_meta.cc:287] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.545s user 0.011s sys 0.001s
I20250905 20:00:08.832283 30736 leader_election.cc:290] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:08.799811 30602 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:08.772670 30762 consensus_queue.cc:1046] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.032s
I20250905 20:00:08.856822 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:08.794835 30732 raft_consensus.cc:513] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:08.860695 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:08.875270 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:08.880602 28082 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 20:00:08.883893 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:08.885020 28076 raft_consensus.cc:2391] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:08.886174 28084 raft_consensus.cc:2391] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 5: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:08.826293 30668 raft_consensus.cc:695] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:08.887959 30602 raft_consensus.cc:491] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:08.907752 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:08.908295 28715 leader_election.cc:336] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 3.295s (SENT)
I20250905 20:00:08.908687 28080 raft_consensus.cc:2466] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 20:00:08.911921 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:08.801733 30759 consensus_queue.cc:1035] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.255s
I20250905 20:00:08.913589 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:08.916141 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 444656us
I20250905 20:00:08.916544 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:08.916829 28049 leader_election.cc:304] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:08.918123 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:08.914566 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 20:00:08.921633 28122 leader_election.cc:336] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.779s (SENT)
I20250905 20:00:08.921887 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:08.922328 28122 leader_election.cc:304] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:08.923008 28082 raft_consensus.cc:2408] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20250905 20:00:08.931509 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:08.930711 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 20:00:08.935585 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:08.931941 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 356819us
I20250905 20:00:08.862375 30540 raft_consensus.cc:513] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:08.985791 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:08.936626 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 439928us
I20250905 20:00:08.994793 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:08.966532 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:08.914168 30762 consensus_queue.cc:1230] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer 7e9c9f591f05464f9b097bf003d14f4a log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20250905 20:00:08.937328 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:08.968225 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:09.009461 28752 raft_consensus.cc:2408] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 6 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250905 20:00:09.017590 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.018682 28085 raft_consensus.cc:2391] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 5: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:09.019649 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.020427 28085 raft_consensus.cc:2466] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 20:00:09.021575 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.022827 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.024003 28085 raft_consensus.cc:2466] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 20:00:09.025669 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.027340 28085 raft_consensus.cc:2466] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 8.
I20250905 20:00:09.028726 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.032783 30717 raft_consensus.cc:491] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:09.032963 30540 leader_election.cc:290] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:09.033367 30717 raft_consensus.cc:513] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:09.033775 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:09.041724 28084 raft_consensus.cc:2466] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 20:00:08.998561 30670 raft_consensus.cc:491] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:08.949046 30773 raft_consensus.cc:2802] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:08.915217 30602 raft_consensus.cc:513] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:08.949435 30772 raft_consensus.cc:2747] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:09.055617 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 20:00:09.059614 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:09.030175 28085 raft_consensus.cc:2466] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6.
I20250905 20:00:09.057914 30773 raft_consensus.cc:491] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:09.035884 30732 leader_election.cc:290] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:09.077584 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:09.056210 30459 raft_consensus.cc:491] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:09.065490 30713 leader_election.cc:290] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:09.034391 30752 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:09.011255 30668 consensus_queue.cc:237] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:09.076191 30773 raft_consensus.cc:3058] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:09.058393 30670 raft_consensus.cc:513] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:09.094556 28752 raft_consensus.cc:1273] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250905 20:00:09.243541 27975 leader_election.cc:304] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:09.330549 28747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40904 (request call id 650) took 2150 ms. Trace:
I20250905 20:00:09.330636 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:09.250245 28010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1463) took 2714 ms. Trace:
I20250905 20:00:09.332723 28751 raft_consensus.cc:2391] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 9: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
W20250905 20:00:09.244385 28009 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1481) took 2339 ms (client timeout 1804 ms). Trace:
W20250905 20:00:09.334604 28009 rpcz_store.cc:269] 0905 20:00:06.904912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:06.910690 (+ 5778us) service_pool.cc:225] Handling call
0905 20:00:09.243477 (+2332787us) spinlock_profiling.cc:243] Waited 823 ms on lock 0x7b54003845d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:09.243992 (+ 515us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1892933888}
I20250905 20:00:09.333007 28010 rpcz_store.cc:276] 0905 20:00:06.536038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:06.536176 (+ 138us) service_pool.cc:225] Handling call
0905 20:00:06.536492 (+ 316us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:06.536703 (+ 211us) raft_consensus.cc:1532] Early marking committed up to index 3
0905 20:00:06.623721 (+ 87018us) log.cc:844] Serialized 14 byte log entry
0905 20:00:09.248623 (+2624902us) spinlock_profiling.cc:243] Waited 3.77 ms on lock 0x7b180041cce0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49d86555c 00007fe49ac873cb 00007fe4a38dcb99 00007fe49aca369b 00007fe49aca193f 00007fe49aca3c02 00007fe49d8572bb 00007fe49d85cdc3 00007fe49d85d2da
0905 20:00:09.249181 (+ 558us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:09.249318 (+ 137us) raft_consensus.cc:1643] Marking committed up to 3
0905 20:00:09.249512 (+ 194us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:09.249698 (+ 186us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:09.249990 (+ 292us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4774,"prepare.queue_time_us":132114,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":4789,"spinlock_wait_cycles":8663936,"thread_start_us":136759,"threads_started":2,"wal-append.queue_time_us":266975,"wal-append.run_cpu_time_us":7988,"wal-append.run_wall_time_us":1351810}
W20250905 20:00:09.238689 27973 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.775s user 0.012s sys 0.010s
W20250905 20:00:09.290465 28122 leader_election.cc:336] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.960s (SENT)
W20250905 20:00:09.289862 28079 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1539) took 2167 ms (client timeout 1552 ms). Trace:
I20250905 20:00:09.287496 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 20:00:09.337849 28079 rpcz_store.cc:269] 0905 20:00:07.122662 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:07.122800 (+ 138us) service_pool.cc:225] Handling call
0905 20:00:09.289336 (+2166536us) spinlock_profiling.cc:243] Waited 340 ms on lock 0x7b540007eed4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:09.289648 (+ 312us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":781622528}
I20250905 20:00:09.339092 28082 raft_consensus.cc:3058] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:09.339411 28744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40904 (request call id 646) took 2257 ms. Trace:
W20250905 20:00:09.339541 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 49359.7us
I20250905 20:00:09.339766 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:09.339974 28744 rpcz_store.cc:276] 0905 20:00:07.081736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:07.081878 (+ 142us) service_pool.cc:225] Handling call
0905 20:00:07.082253 (+ 375us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:07.242319 (+160066us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:07.244163 (+ 1844us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:07.251446 (+ 7283us) raft_consensus.cc:1643] Marking committed up to 6
0905 20:00:07.256637 (+ 5191us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:09.338650 (+2082013us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b54004b0d54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:09.338743 (+ 93us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:09.339155 (+ 412us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":291761920}
W20250905 20:00:09.337265 27973 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.874s user 0.012s sys 0.011s
I20250905 20:00:09.301319 28083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1554) took 2119 ms. Trace:
W20250905 20:00:09.341655 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 878931us
I20250905 20:00:09.284058 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.290894 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:09.328581 28749 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.006s sys 0.011s
I20250905 20:00:09.285825 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:09.344162 28081 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1549) took 2164 ms (client timeout 2530 ms). Trace:
W20250905 20:00:09.344663 28081 rpcz_store.cc:269] 0905 20:00:07.180087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:09.290371 (+2110284us) service_pool.cc:225] Handling call
0905 20:00:09.344071 (+ 53700us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 20:00:09.344960 28077 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1553) took 2163 ms (client timeout 2680 ms). Trace:
W20250905 20:00:09.345443 28077 rpcz_store.cc:269] 0905 20:00:07.181247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:09.285181 (+2103934us) service_pool.cc:225] Handling call
0905 20:00:09.344903 (+ 59722us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:09.345742 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:09.340871 28122 leader_election.cc:336] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 2.013s (SENT)
I20250905 20:00:09.315517 28155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39586 (request call id 305) took 3692 ms. Trace:
I20250905 20:00:09.346721 28122 leader_election.cc:304] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:09.343753 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 20:00:09.316591 28159 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39586 (request call id 300) took 3695 ms (client timeout 3214 ms). Trace:
I20250905 20:00:09.289096 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:09.347536 28159 rpcz_store.cc:269] 0905 20:00:05.620929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:06.009561 (+388632us) service_pool.cc:225] Handling call
0905 20:00:09.316101 (+3306540us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b54000e1ad4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:09.316347 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3198700800}
I20250905 20:00:09.348307 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.348588 28085 raft_consensus.cc:2466] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:09.349689 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.335803 28745 raft_consensus.cc:3058] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:09.331355 28747 rpcz_store.cc:276] 0905 20:00:07.180413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:07.192336 (+ 11923us) service_pool.cc:225] Handling call
0905 20:00:07.240679 (+ 48343us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:08.134614 (+893935us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:08.135010 (+ 396us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:08.135644 (+ 634us) log.cc:844] Serialized 26 byte log entry
0905 20:00:08.309549 (+173905us) raft_consensus.cc:1643] Marking committed up to 6
0905 20:00:08.310406 (+ 857us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:09.329589 (+1019183us) spinlock_profiling.cc:243] Waited 887 ms on lock 0x7b5400354b54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:09.329667 (+ 78us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:09.329803 (+ 136us) raft_consensus.cc:1683] finished
0905 20:00:09.329818 (+ 15us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:09.330333 (+ 515us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5117,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":82,"spinlock_wait_cycles":2039162752,"thread_start_us":173448,"threads_started":1,"wal-append.queue_time_us":269333}
W20250905 20:00:09.343935 28749 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.557s user 0.007s sys 0.011s
I20250905 20:00:09.341904 28083 rpcz_store.cc:276] 0905 20:00:07.181526 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:08.265226 (+1083700us) service_pool.cc:225] Handling call
0905 20:00:08.265562 (+ 336us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:08.297246 (+ 31684us) raft_consensus.cc:1532] Early marking committed up to index 4
0905 20:00:08.440143 (+142897us) log.cc:844] Serialized 14 byte log entry
0905 20:00:08.521098 (+ 80955us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:08.521170 (+ 72us) raft_consensus.cc:1643] Marking committed up to 4
0905 20:00:08.521234 (+ 64us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:09.300563 (+779329us) spinlock_profiling.cc:243] Waited 31.3 ms on lock 0x7b54001f04d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:09.300657 (+ 94us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:09.301037 (+ 380us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24908,"prepare.queue_time_us":164550,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":65107,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":9157,"spinlock_wait_cycles":72071040,"thread_start_us":196117,"threads_started":2,"wal-append.queue_time_us":349190}
I20250905 20:00:09.328743 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:09.353474 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:09.332013 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 88957.4us
I20250905 20:00:09.354862 28743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40926 (request call id 486) took 1239 ms. Trace:
I20250905 20:00:09.349835 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.355314 28743 rpcz_store.cc:276] 0905 20:00:08.115654 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:08.116865 (+ 1211us) service_pool.cc:225] Handling call
0905 20:00:09.354329 (+1237464us) spinlock_profiling.cc:243] Waited 198 ms on lock 0x7b54004e07d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:09.354609 (+ 280us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":456308096}
I20250905 20:00:09.356007 27975 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:09.356546 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:09.325024 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:09.341290 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.358888 28079 raft_consensus.cc:2391] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:09.346495 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.342962 28748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40918 (request call id 563) took 2717 ms. Trace:
I20250905 20:00:09.360688 28748 rpcz_store.cc:276] 0905 20:00:06.625278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:06.625426 (+ 148us) service_pool.cc:225] Handling call
0905 20:00:06.625753 (+ 327us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:07.461399 (+835646us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:09.342379 (+1880980us) spinlock_profiling.cc:243] Waited 832 ms on lock 0x7b540053cd54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:09.342753 (+ 374us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1913717632}
I20250905 20:00:09.337270 28750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40904 (request call id 640) took 3511 ms. Trace:
I20250905 20:00:09.362197 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:09.312960 28154 raft_consensus.cc:3053] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 20:00:09.362547 28750 rpcz_store.cc:276] 0905 20:00:05.826081 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:05.826482 (+ 401us) service_pool.cc:225] Handling call
0905 20:00:05.826890 (+ 408us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:06.588310 (+761420us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:09.336598 (+2748288us) spinlock_profiling.cc:243] Waited 760 ms on lock 0x7b54001568d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:09.337065 (+ 467us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1749055744}
I20250905 20:00:09.362991 28085 raft_consensus.cc:2391] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 5: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:09.363242 28076 raft_consensus.cc:2391] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 5: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 20:00:09.363860 28154 raft_consensus.cc:738] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
W20250905 20:00:09.323107 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:09.346976 28155 rpcz_store.cc:276] 0905 20:00:05.622555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:06.169583 (+547028us) service_pool.cc:225] Handling call
0905 20:00:06.257267 (+ 87684us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:08.114047 (+1856780us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:09.314111 (+1200064us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b540015fed4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:09.315044 (+ 933us) spinlock_profiling.cc:243] Waited 87.1 ms on lock 0x7b540015fed0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:09.315287 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3446228736}
I20250905 20:00:09.099345 30459 raft_consensus.cc:513] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:09.098554 30470 leader_election.cc:290] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:09.360071 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.375581 28079 raft_consensus.cc:2466] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:09.376317 30263 raft_consensus.cc:2802] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 20:00:09.376791 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.377734 28079 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 20:00:09.378041 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.378131 30517 raft_consensus.cc:491] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:09.378991 30517 raft_consensus.cc:3058] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:09.379987 30402 consensus_queue.cc:1035] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 20:00:09.365721 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.343636 27973 leader_election.cc:304] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:09.379055 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:09.385259 28079 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:09.385710 28154 consensus_queue.cc:260] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:09.376801 30263 raft_consensus.cc:491] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:09.382470 30402 raft_consensus.cc:2747] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:09.387046 28154 raft_consensus.cc:3058] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:09.387315 30263 raft_consensus.cc:3058] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:09.382229 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.389863 28076 raft_consensus.cc:2366] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5.
I20250905 20:00:09.382184 30374 raft_consensus.cc:2747] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250905 20:00:09.392292 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 48916.8us
I20250905 20:00:09.392786 30148 raft_consensus.cc:2802] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 20:00:09.393682 30148 raft_consensus.cc:695] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:09.378450 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.395323 28085 raft_consensus.cc:2366] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5.
I20250905 20:00:09.396256 30148 consensus_queue.cc:237] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:09.399664 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:09.396896 28081 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
W20250905 20:00:09.398022 30537 consensus_meta.cc:287] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.675s user 0.006s sys 0.009s
I20250905 20:00:09.355332 28083 raft_consensus.cc:2391] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
W20250905 20:00:09.409545 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 44405.7us
W20250905 20:00:09.425746 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:09.432757 28085 raft_consensus.cc:3058] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:09.441330 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.441705 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:09.444675 28081 raft_consensus.cc:3058] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:09.451845 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:09.169566 30707 log.cc:927] Time spent T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 2.293s user 0.013s sys 0.012s
W20250905 20:00:09.168216 30719 log.cc:927] Time spent T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 2.087s user 0.004s sys 0.011s
I20250905 20:00:09.465000 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:09.203626 30773 raft_consensus.cc:513] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:09.145577 30752 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 9 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:09.476559 28081 raft_consensus.cc:2466] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
I20250905 20:00:09.158810 30756 raft_consensus.cc:3053] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 LEADER]: Stepping down as leader of term 6
I20250905 20:00:09.479251 28080 raft_consensus.cc:2391] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 6: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:09.480628 30663 raft_consensus.cc:3053] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 20:00:09.481200 30663 raft_consensus.cc:738] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:09.143666 30717 leader_election.cc:290] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:09.490875 27975 leader_election.cc:304] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:09.492478 30682 raft_consensus.cc:2802] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 20:00:09.493098 30682 raft_consensus.cc:491] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:09.493386 27975 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:09.493676 30682 raft_consensus.cc:3058] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:09.494647 30709 raft_consensus.cc:491] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:09.495136 28078 raft_consensus.cc:1273] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250905 20:00:09.495464 30709 raft_consensus.cc:513] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:09.496757 30459 leader_election.cc:290] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:09.497505 28078 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1555) took 2315 ms. Trace:
I20250905 20:00:09.505982 30543 raft_consensus.cc:2802] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250905 20:00:09.508423 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:09.508361 30543 raft_consensus.cc:491] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:09.511998 30602 leader_election.cc:290] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:09.512317 30670 leader_election.cc:290] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:09.518826 28083 raft_consensus.cc:2466] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 20:00:09.439982 30537 raft_consensus.cc:513] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:09.519937 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:09.523948 30741 raft_consensus.cc:491] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:09.524492 30537 leader_election.cc:290] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:09.524868 30741 raft_consensus.cc:513] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:09.527669 30635 consensus_meta.cc:287] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.914s user 0.010s sys 0.002s
I20250905 20:00:09.528306 30635 raft_consensus.cc:513] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:09.530856 30543 raft_consensus.cc:3058] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
W20250905 20:00:09.536217 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:09.469208 30690 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.903s user 0.008s sys 0.029s
I20250905 20:00:09.536877 30672 raft_consensus.cc:491] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
W20250905 20:00:09.537379 30690 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 3.007s user 0.008s sys 0.030s
I20250905 20:00:09.537815 30690 raft_consensus.cc:695] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:09.539623 30690 consensus_queue.cc:237] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:09.540570 30675 consensus_meta.cc:287] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 1.204s user 0.009s sys 0.007s
I20250905 20:00:09.494699 30779 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:09.543052 30618 leader_election.cc:290] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:09.537935 30672 raft_consensus.cc:513] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:09.147516 30755 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 20:00:09.546028 28049 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.014s sys 0.018s
W20250905 20:00:09.546519 28049 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.610s user 0.014s sys 0.018s
I20250905 20:00:09.545337 30612 raft_consensus.cc:3053] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Stepping down as leader of term 4
I20250905 20:00:09.547667 30612 raft_consensus.cc:738] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:09.520061 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.553258 28079 raft_consensus.cc:2466] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 8.
I20250905 20:00:09.554216 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.497908 28078 rpcz_store.cc:276] 0905 20:00:07.181808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:09.286593 (+2104785us) service_pool.cc:225] Handling call
0905 20:00:09.287359 (+ 766us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:09.495558 (+208199us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:09.497218 (+ 1660us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b54003a9654. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:09.497421 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":368745344}
I20250905 20:00:09.568099 28082 raft_consensus.cc:2466] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:09.571590 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.581508 28076 raft_consensus.cc:2391] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:09.584357 30756 raft_consensus.cc:738] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:09.585664 30756 consensus_queue.cc:260] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:09.587270 30756 raft_consensus.cc:3058] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Advancing to term 7
W20250905 20:00:09.587661 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:09.588213 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:09.588558 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:09.589039 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
W20250905 20:00:09.589460 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:09.590538 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:09.591578 28749 raft_consensus.cc:2408] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250905 20:00:09.591200 30517 raft_consensus.cc:513] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:09.580549 28154 raft_consensus.cc:1273] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250905 20:00:09.550513 30709 leader_election.cc:290] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:09.582502 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.595701 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.596873 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:09.598341 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 559466us
I20250905 20:00:09.598635 30542 consensus_queue.cc:1035] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20250905 20:00:09.599052 30739 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20250905 20:00:09.599359 28745 raft_consensus.cc:1273] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250905 20:00:09.622757 28154 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1617) took 1373 ms. Trace:
I20250905 20:00:09.623287 28154 rpcz_store.cc:276] 0905 20:00:08.249218 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:08.345268 (+ 96050us) service_pool.cc:225] Handling call
0905 20:00:08.345535 (+ 267us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:09.312635 (+967100us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b5400070920. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dba8550 00007fe49dc1b170 00007fe49dc1c1e1 00007fe49dc18d4f 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c
0905 20:00:09.592845 (+280210us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:09.622655 (+ 29810us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":251902336}
I20250905 20:00:09.624917 28078 raft_consensus.cc:2391] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 6: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:09.626096 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.541086 30675 raft_consensus.cc:513] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:09.611063 28122 leader_election.cc:336] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 2.166s (SENT)
W20250905 20:00:09.687573 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 86146.7us
I20250905 20:00:09.626855 28078 raft_consensus.cc:2466] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 20:00:09.698494 28156 raft_consensus.cc:1238] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:09.676430 28745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40904 (request call id 629) took 4923 ms. Trace:
W20250905 20:00:09.676293 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 60393.2us
W20250905 20:00:09.729646 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 236402us
I20250905 20:00:09.758824 28745 rpcz_store.cc:276] 0905 20:00:04.753049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:04.759540 (+ 6491us) service_pool.cc:225] Handling call
0905 20:00:04.759883 (+ 343us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:05.764599 (+1004716us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b54004b08a0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dba8550 00007fe49dc1b170 00007fe49dc1c1e1 00007fe49dc18d4f 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c
0905 20:00:09.333947 (+3569348us) spinlock_profiling.cc:243] Waited 905 ms on lock 0x7b580079ebb0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49db94003 00007fe49dbf7577 00007fe49dc0ffa7 00007fe49dc1179e 00007fe49dc1baf9 00007fe49dc1c3c6 00007fe49dc18d4f 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44
0905 20:00:09.615469 (+281522us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:09.676229 (+ 60760us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2709057280}
I20250905 20:00:09.668340 28083 raft_consensus.cc:2466] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 8.
I20250905 20:00:09.671283 28155 raft_consensus.cc:1238] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:09.730528 28081 raft_consensus.cc:1238] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250905 20:00:09.675861 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.697521 28153 raft_consensus.cc:1238] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:09.763512 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.764495 28079 raft_consensus.cc:2391] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 7: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:09.662497 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:09.626788 30597 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
W20250905 20:00:09.681587 30788 log.cc:927] Time spent T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.080s user 0.001s sys 0.003s
W20250905 20:00:09.722189 30798 log.cc:927] Time spent T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.121s user 0.000s sys 0.000s
I20250905 20:00:09.671880 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:09.770509 28085 raft_consensus.cc:2466] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 20:00:09.769074 28122 leader_election.cc:304] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:09.595039 30672 leader_election.cc:290] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:09.699903 30674 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.727s user 0.000s sys 0.028s
W20250905 20:00:09.668020 30552 consensus_meta.cc:287] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.214s user 0.008s sys 0.007s
I20250905 20:00:09.773684 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 20:00:09.770004 27973 connection.cc:232] RPC call timeout handler was delayed by -1.6331355040000517s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1552s for a response
I20250905 20:00:09.785843 28083 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Advancing to term 9
I20250905 20:00:09.694744 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:09.686676 30668 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.566s user 0.018s sys 0.009s
I20250905 20:00:09.775724 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.690492 30517 leader_election.cc:290] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:09.715692 30779 raft_consensus.cc:2747] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250905 20:00:09.779224 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:09.730005 30619 raft_consensus.cc:3058] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:09.660180 30773 leader_election.cc:290] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:09.687920 30670 consensus_queue.cc:1046] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.111s
I20250905 20:00:09.675922 30635 leader_election.cc:290] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:09.670641 30537 raft_consensus.cc:491] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:09.779347 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.670218 30772 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:09.675568 30741 leader_election.cc:290] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:09.666175 30588 raft_consensus.cc:491] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:09.792974 27975 leader_election.cc:304] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
W20250905 20:00:09.785027 30674 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.915s user 0.005s sys 0.028s
I20250905 20:00:09.628180 30594 raft_consensus.cc:3058] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:09.819002 30721 raft_consensus.cc:491] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:09.825016 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:09.828614 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:09.628710 30682 raft_consensus.cc:513] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:09.674583 30596 leader_election.cc:290] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:09.664037 30263 raft_consensus.cc:513] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:09.846235 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 56098.5us
I20250905 20:00:09.604426 30663 consensus_queue.cc:260] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:09.677956 30628 consensus_queue.cc:1035] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.180s
I20250905 20:00:09.900547 28082 raft_consensus.cc:1238] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250905 20:00:09.943521 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:09.667212 30612 consensus_queue.cc:260] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:09.870594 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:09.866245 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.953398 28081 raft_consensus.cc:1238] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 3. Current term is 5. Ops: []
I20250905 20:00:09.966317 28081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 280) took 1155 ms. Trace:
I20250905 20:00:09.962183 28082 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1499) took 1105 ms. Trace:
W20250905 20:00:09.962459 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 103332us
I20250905 20:00:09.970857 30263 leader_election.cc:290] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:09.964491 28076 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 20:00:09.967231 28082 rpcz_store.cc:276] 0905 20:00:08.856581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:09.858009 (+1001428us) service_pool.cc:225] Handling call
0905 20:00:09.900250 (+ 42241us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:09.961872 (+ 61622us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:09.962114 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:09.973852 28076 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50660 (request call id 285) took 1161 ms. Trace:
I20250905 20:00:09.966687 28081 rpcz_store.cc:276] 0905 20:00:08.810792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:09.811631 (+1000839us) service_pool.cc:225] Handling call
0905 20:00:09.952362 (+140731us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:09.966050 (+ 13688us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:09.966260 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 20:00:09.847242 30668 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.727s user 0.019s sys 0.015s
I20250905 20:00:09.977478 28083 raft_consensus.cc:2466] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 9.
I20250905 20:00:09.974416 28076 rpcz_store.cc:276] 0905 20:00:08.812123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:09.607996 (+795873us) service_pool.cc:225] Handling call
0905 20:00:09.973775 (+365779us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 20:00:09.979765 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:09.980067 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:09.978953 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:09.981031 28083 raft_consensus.cc:2374] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 8. Re-sending same reply.
W20250905 20:00:09.944271 30684 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.655s user 0.006s sys 0.004s
W20250905 20:00:09.983484 30684 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.849s user 0.006s sys 0.006s
I20250905 20:00:09.984043 30612 raft_consensus.cc:3058] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:09.984606 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:09.985345 28156 raft_consensus.cc:3058] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:09.986552 28155 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 20:00:09.987846 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:09.988822 30663 raft_consensus.cc:3058] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:09.811429 30552 raft_consensus.cc:513] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:09.991194 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:09.993010 30552 leader_election.cc:290] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:09.998883 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:09.999073 30393 raft_consensus.cc:491] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:09.999610 28744 raft_consensus.cc:1273] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250905 20:00:09.999989 30393 raft_consensus.cc:513] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:10.000913 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:10.001843 28158 raft_consensus.cc:1238] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: []
W20250905 20:00:10.003345 30800 log.cc:927] Time spent T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20250905 20:00:09.978520 30721 raft_consensus.cc:513] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:10.004781 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:10.005537 28010 raft_consensus.cc:2374] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6. Re-sending same reply.
I20250905 20:00:10.005244 30588 raft_consensus.cc:513] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:10.006887 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:10.007313 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:10.007647 30588 leader_election.cc:290] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:10.011377 30660 consensus_meta.cc:287] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 1.368s user 0.016s sys 0.000s
I20250905 20:00:10.017243 30721 leader_election.cc:290] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:10.019683 28085 raft_consensus.cc:1238] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250905 20:00:10.020437 28085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1496) took 1164 ms. Trace:
W20250905 20:00:10.022379 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 439404us
I20250905 20:00:10.020902 28085 rpcz_store.cc:276] 0905 20:00:08.855606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:10.018921 (+1163315us) service_pool.cc:225] Handling call
0905 20:00:10.019336 (+ 415us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:10.020141 (+ 805us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:10.020357 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:09.943322 30777 raft_consensus.cc:491] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:10.027127 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:10.029944 28082 raft_consensus.cc:2391] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 5: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 20:00:10.028075 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:10.030758 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 49563us
I20250905 20:00:10.031275 28156 raft_consensus.cc:2466] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6.
I20250905 20:00:10.031289 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:10.036339 28078 raft_consensus.cc:2391] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 5: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
W20250905 20:00:10.036484 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:10.037982 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:09.920776 30739 raft_consensus.cc:2747] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250905 20:00:10.040266 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:10.040907 28085 raft_consensus.cc:2374] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7. Re-sending same reply.
I20250905 20:00:10.042542 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:10.043519 28083 raft_consensus.cc:2391] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 5: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
W20250905 20:00:10.044281 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:10.048009 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:10.048382 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:09.955525 30543 raft_consensus.cc:513] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:10.050673 28122 leader_election.cc:336] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 2.172s (SENT)
I20250905 20:00:10.050639 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:10.027318 30393 leader_election.cc:290] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:10.054817 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:10.057950 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:10.058704 28076 raft_consensus.cc:2391] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 9: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:10.060616 28079 raft_consensus.cc:1238] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 4. Current term is 5. Ops: []
W20250905 20:00:10.062706 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:09.980576 30148 consensus_queue.cc:1035] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.073s
I20250905 20:00:10.065001 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:10.066352 28079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 276) took 1252 ms. Trace:
I20250905 20:00:10.025889 30537 raft_consensus.cc:513] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:10.066996 28079 rpcz_store.cc:276] 0905 20:00:08.808680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:09.768938 (+960258us) service_pool.cc:225] Handling call
0905 20:00:09.979463 (+210525us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:10.061165 (+ 81702us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:10.061413 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:10.070974 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:10.071691 28083 raft_consensus.cc:2466] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:10.072995 30660 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:10.073557 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:10.070219 30684 raft_consensus.cc:2747] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 20:00:10.073103 28009 raft_consensus.cc:1238] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 3. Current term is 4. Ops: []
I20250905 20:00:10.075842 28010 raft_consensus.cc:2374] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4. Re-sending same reply.
I20250905 20:00:10.076171 28156 raft_consensus.cc:2391] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 4: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:10.077769 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:10.078701 28076 raft_consensus.cc:2391] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 6: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:10.080536 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:10.081226 28083 raft_consensus.cc:2391] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 6: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:10.067754 30675 leader_election.cc:290] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:10.087404 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:10.039388 30772 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:10.090514 28078 raft_consensus.cc:1238] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 3. Current term is 4. Ops: []
W20250905 20:00:10.093636 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 55681.9us
I20250905 20:00:10.080286 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:10.073920 30660 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:10.097049 28081 raft_consensus.cc:1238] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20250905 20:00:10.097131 28085 raft_consensus.cc:2466] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
I20250905 20:00:10.100775 28078 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1502) took 1096 ms. Trace:
I20250905 20:00:10.101393 28078 rpcz_store.cc:276] 0905 20:00:08.999111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:10.036912 (+1037801us) service_pool.cc:225] Handling call
0905 20:00:10.070639 (+ 33727us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:10.093788 (+ 23149us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:10.093983 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:10.104405 28081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1503) took 1088 ms. Trace:
I20250905 20:00:10.104812 28081 rpcz_store.cc:276] 0905 20:00:09.016065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:10.052883 (+1036818us) service_pool.cc:225] Handling call
0905 20:00:10.083753 (+ 30870us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:10.104170 (+ 20417us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:10.104357 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:10.089963 30682 leader_election.cc:290] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:10.106247 30543 leader_election.cc:290] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:10.106557 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:10.112869 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 341266us
I20250905 20:00:10.060163 30777 raft_consensus.cc:513] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:10.124595 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:10.121572 28715 leader_election.cc:336] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 3.214s (SENT)
I20250905 20:00:10.121807 28082 raft_consensus.cc:1238] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 3. Current term is 4. Ops: []
I20250905 20:00:10.128485 28082 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1504) took 1101 ms. Trace:
I20250905 20:00:10.128736 28082 rpcz_store.cc:276] 0905 20:00:09.026912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:10.062488 (+1035576us) service_pool.cc:225] Handling call
0905 20:00:10.097719 (+ 35231us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:10.128227 (+ 30508us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:10.128427 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:10.131685 30772 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:10.132995 28122 leader_election.cc:336] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 2.179s (SENT)
I20250905 20:00:10.133682 28085 raft_consensus.cc:1238] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250905 20:00:10.135957 28078 raft_consensus.cc:1238] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20250905 20:00:10.136844 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:10.095672 30684 consensus_queue.cc:1035] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.029s
I20250905 20:00:10.139293 28078 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1509) took 1110 ms. Trace:
I20250905 20:00:10.140475 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:10.143507 28078 rpcz_store.cc:276] 0905 20:00:09.028839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:10.102391 (+1073552us) service_pool.cc:225] Handling call
0905 20:00:10.134239 (+ 31848us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:10.139025 (+ 4786us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:10.139238 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:10.141882 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:10.137070 28085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1508) took 1108 ms. Trace:
I20250905 20:00:10.146190 28156 raft_consensus.cc:1273] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250905 20:00:10.146461 28081 raft_consensus.cc:1238] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:10.146099 28085 rpcz_store.cc:276] 0905 20:00:09.028466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:10.100440 (+1071974us) service_pool.cc:225] Handling call
0905 20:00:10.133453 (+ 33013us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:10.136442 (+ 2989us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:10.136992 (+ 550us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:10.148079 28079 raft_consensus.cc:1238] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:10.152042 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:10.152259 28079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1507) took 1124 ms. Trace:
I20250905 20:00:10.152755 28079 rpcz_store.cc:276] 0905 20:00:09.028125 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:10.095857 (+1067732us) service_pool.cc:225] Handling call
0905 20:00:10.134582 (+ 38725us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:10.152011 (+ 17429us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:10.152205 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:10.154706 28081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1510) took 1125 ms. Trace:
I20250905 20:00:10.155145 28081 rpcz_store.cc:276] 0905 20:00:09.029164 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:10.105381 (+1076217us) service_pool.cc:225] Handling call
0905 20:00:10.146243 (+ 40862us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:10.153983 (+ 7740us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:10.154650 (+ 667us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":56320}
W20250905 20:00:10.153132 28715 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.728s (SENT) [suppressed 38 similar messages]
I20250905 20:00:10.164754 28079 raft_consensus.cc:1273] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250905 20:00:10.162981 28750 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 20:00:10.164386 30559 raft_consensus.cc:491] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:10.164798 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:10.171320 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:10.115721 28077 raft_consensus.cc:2466] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 20:00:10.178387 28150 raft_consensus.cc:2408] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250905 20:00:10.172659 28151 raft_consensus.cc:2466] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6.
I20250905 20:00:10.166508 28123 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:10.169926 28078 raft_consensus.cc:1238] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 3. Current term is 4. Ops: []
I20250905 20:00:10.181154 28750 pending_rounds.cc:85] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a: Aborting all ops after (but not including) 4
I20250905 20:00:10.181519 28750 pending_rounds.cc:107] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20250905 20:00:10.184589 28750 raft_consensus.cc:2887] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250905 20:00:10.184866 28078 raft_consensus.cc:1238] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:10.187234 28081 raft_consensus.cc:1273] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250905 20:00:10.195920 28077 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1512) took 1166 ms. Trace:
I20250905 20:00:10.196429 28077 rpcz_store.cc:276] 0905 20:00:09.029822 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:09.361760 (+331938us) service_pool.cc:225] Handling call
0905 20:00:10.195705 (+833945us) spinlock_profiling.cc:243] Waited 648 ms on lock 0x7b540019f5d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:10.195844 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1491051264}
I20250905 20:00:10.197605 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 20:00:10.199805 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 72764.8us
W20250905 20:00:10.184902 30803 log.cc:927] Time spent T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410: Append to log took a long time: real 0.058s user 0.000s sys 0.001s
I20250905 20:00:10.208572 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:10.209487 28151 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 20:00:10.206101 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:10.211212 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:10.218233 30670 consensus_queue.cc:1046] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250905 20:00:10.220021 30670 raft_consensus.cc:3053] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 LEADER]: Stepping down as leader of term 5
I20250905 20:00:10.220441 30670 raft_consensus.cc:738] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:10.221468 27975 leader_election.cc:304] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
W20250905 20:00:10.221485 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:10.218710 28010 raft_consensus.cc:1215] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Deduplicated request from leader. Original: 1.1->[2.2-4.3] Dedup: 2.2->[4.3-4.3]
I20250905 20:00:10.223028 28010 raft_consensus.cc:1238] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20250905 20:00:10.225143 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:10.225625 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:10.204537 30537 leader_election.cc:290] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:10.192843 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:10.236934 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:10.237905 28154 raft_consensus.cc:2466] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 20:00:10.240226 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:10.244402 28158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2153) took 1521 ms. Trace:
I20250905 20:00:10.244926 28158 rpcz_store.cc:276] 0905 20:00:08.723137 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:08.723273 (+ 136us) service_pool.cc:225] Handling call
0905 20:00:08.723678 (+ 405us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:10.002313 (+1278635us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:10.242001 (+239688us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b5400327654. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:10.244328 (+ 2327us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2938454272}
I20250905 20:00:10.181733 30559 raft_consensus.cc:513] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:10.256863 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 136250us
I20250905 20:00:10.248652 30675 raft_consensus.cc:491] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:10.262528 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:10.263587 28010 raft_consensus.cc:2391] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 9: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:10.263181 30675 raft_consensus.cc:513] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:10.265925 30675 leader_election.cc:290] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:10.268828 28002 raft_consensus.cc:1238] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 3. Current term is 4. Ops: []
I20250905 20:00:10.269306 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:10.270535 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:10.271437 28006 raft_consensus.cc:1238] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:10.241899 30690 raft_consensus.cc:2747] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250905 20:00:10.294090 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 72714.5us
I20250905 20:00:10.250981 30635 consensus_queue.cc:1046] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.055s
I20250905 20:00:10.222705 30670 consensus_queue.cc:260] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:10.311862 30635 raft_consensus.cc:3053] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 LEADER]: Stepping down as leader of term 5
W20250905 20:00:10.316504 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:10.316630 30635 raft_consensus.cc:738] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
W20250905 20:00:10.317543 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:10.317606 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:10.318769 28717 leader_election.cc:336] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Timed out: RequestConsensusVote RPC to 127.26.58.131:36705 timed out after 3.214s (SENT)
I20250905 20:00:10.319267 28717 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:10.321131 30612 raft_consensus.cc:491] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:10.322010 30612 raft_consensus.cc:513] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:10.324661 30612 leader_election.cc:290] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 20:00:10.308043 30805 log.cc:927] Time spent T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.195s user 0.000s sys 0.001s
I20250905 20:00:10.326191 30612 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:10.345602 28010 raft_consensus.cc:1238] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 4. Current term is 5. Ops: []
W20250905 20:00:10.352710 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:10.360812 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 20:00:10.367377 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:10.369438 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:10.370020 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:10.370802 28158 raft_consensus.cc:2466] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
I20250905 20:00:10.370637 28150 raft_consensus.cc:2466] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
I20250905 20:00:10.372108 28717 leader_election.cc:304] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:10.343072 30777 leader_election.cc:290] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:10.374316 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:10.375550 28003 raft_consensus.cc:2391] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 9: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:10.377679 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:10.378157 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:10.340546 30721 raft_consensus.cc:491] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:10.379309 28006 raft_consensus.cc:1238] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:10.378988 30721 raft_consensus.cc:513] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:10.379746 30612 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:10.343516 28077 raft_consensus.cc:1273] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250905 20:00:10.382535 27975 leader_election.cc:304] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:10.359745 28159 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 20:00:10.317709 30635 consensus_queue.cc:260] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:10.404699 28050 leader_election.cc:304] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:10.407503 28050 leader_election.cc:304] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:10.416935 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:10.418869 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:10.419795 30775 raft_consensus.cc:2802] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250905 20:00:10.420318 30804 log.cc:927] Time spent T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.267s user 0.001s sys 0.001s
I20250905 20:00:10.420040 30672 raft_consensus.cc:2802] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:10.397423 30660 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:10.422712 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:10.406535 30663 raft_consensus.cc:2802] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:10.413563 30777 raft_consensus.cc:2802] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 20:00:10.425777 30721 leader_election.cc:290] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:10.437273 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 123865us
I20250905 20:00:10.429083 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:10.400292 28005 raft_consensus.cc:2466] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 20:00:10.431246 28003 raft_consensus.cc:1273] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250905 20:00:10.414604 30670 raft_consensus.cc:3058] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:10.426198 30775 raft_consensus.cc:491] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:10.340209 30725 consensus_queue.cc:1046] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.063s
I20250905 20:00:10.426004 30672 raft_consensus.cc:491] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:10.428324 30741 raft_consensus.cc:2802] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader election won for term 6
W20250905 20:00:10.452454 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:10.453022 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:10.432802 30777 raft_consensus.cc:491] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:10.450496 30635 raft_consensus.cc:3058] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:10.463135 30660 raft_consensus.cc:3053] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 20:00:10.463620 30660 raft_consensus.cc:738] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:10.445348 30597 raft_consensus.cc:2747] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:10.446107 30775 raft_consensus.cc:3058] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:10.443516 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:10.453310 30672 raft_consensus.cc:3058] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:10.473012 27975 leader_election.cc:304] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:10.460562 30777 raft_consensus.cc:3058] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
W20250905 20:00:10.480427 28122 leader_election.cc:336] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.590s (SENT)
I20250905 20:00:10.472247 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:10.485778 28122 leader_election.cc:304] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:10.481406 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:10.490743 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:10.495052 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:10.497646 28152 raft_consensus.cc:604] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Received request to transfer leadership to TS e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:10.498183 28152 raft_consensus.cc:612] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250905 20:00:10.506322 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 59039.5us
W20250905 20:00:10.511869 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:10.501223 28746 raft_consensus.cc:2408] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20250905 20:00:10.513837 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:10.472613 30660 consensus_queue.cc:260] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:10.501291 30594 raft_consensus.cc:2747] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:10.532649 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:10.540463 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:10.541260 28077 raft_consensus.cc:2466] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:10.541980 30721 consensus_queue.cc:1046] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250905 20:00:10.543145 30721 raft_consensus.cc:3053] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Stepping down as leader of term 4
I20250905 20:00:10.543499 30721 raft_consensus.cc:738] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:10.544397 30721 consensus_queue.cc:260] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:10.545630 30721 raft_consensus.cc:3058] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:10.546553 30683 consensus_queue.cc:1035] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.029s
W20250905 20:00:10.547552 30806 log.cc:927] Time spent T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.053s user 0.000s sys 0.001s
I20250905 20:00:10.555403 28085 raft_consensus.cc:1238] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6. Ops: []
W20250905 20:00:10.556784 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:10.557602 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:10.565811 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:10.572230 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:10.576254 28122 leader_election.cc:336] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.645s (SENT)
I20250905 20:00:10.576884 28122 leader_election.cc:304] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:10.575555 28049 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:10.566845 28077 raft_consensus.cc:1273] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250905 20:00:10.579066 28718 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:10.580889 30588 consensus_queue.cc:1046] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.006s
I20250905 20:00:10.587785 30756 raft_consensus.cc:3053] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 LEADER]: Stepping down as leader of term 7
I20250905 20:00:10.588256 30756 raft_consensus.cc:738] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:10.588411 28152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:36164 (request call id 22) took 2145 ms. Trace:
I20250905 20:00:10.588856 28152 rpcz_store.cc:276] 0905 20:00:08.442611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:08.442730 (+ 119us) service_pool.cc:225] Handling call
0905 20:00:10.588143 (+2145413us) spinlock_profiling.cc:243] Waited 1.98 s on lock 0x7b5400185e54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc13813 00007fe4a2b801ca 00007fe49c2131c4 00007fe49c213152 00007fe49c213081 00007fe49c212ffc 00007fe49c2122b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:10.588323 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4555453824}
I20250905 20:00:10.589339 30756 consensus_queue.cc:260] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:10.591041 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:10.567627 30612 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:10.602552 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:10.609380 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:10.594970 30619 raft_consensus.cc:2747] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:10.609158 30660 raft_consensus.cc:3058] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
W20250905 20:00:10.620795 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:10.601420 30668 raft_consensus.cc:2747] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 20:00:10.618412 30240 raft_consensus.cc:491] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:10.654178 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 62942.6us
I20250905 20:00:10.646075 28080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 278) took 1836 ms. Trace:
W20250905 20:00:10.661280 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 95680.1us
I20250905 20:00:10.676280 28076 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1505) took 1648 ms. Trace:
I20250905 20:00:10.676571 30597 raft_consensus.cc:2747] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20250905 20:00:10.677955 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:10.681442 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 110113us
I20250905 20:00:10.679955 28083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1506) took 1652 ms. Trace:
I20250905 20:00:10.677495 28076 rpcz_store.cc:276] 0905 20:00:09.027437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:10.084552 (+1057115us) service_pool.cc:225] Handling call
0905 20:00:10.122993 (+ 38441us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:10.159797 (+ 36804us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:10.159839 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:10.213573 (+ 53734us) log.cc:844] Serialized 26 byte log entry
0905 20:00:10.571821 (+358248us) raft_consensus.cc:1643] Marking committed up to 2
0905 20:00:10.571924 (+ 103us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:10.571994 (+ 70us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:10.676037 (+104043us) raft_consensus.cc:1683] finished
0905 20:00:10.676060 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:10.676215 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2863,"raft.queue_time_us":9819,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":140,"thread_start_us":313821,"threads_started":1,"wal-append.queue_time_us":442871}
I20250905 20:00:10.686681 28083 rpcz_store.cc:276] 0905 20:00:09.027798 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:10.081672 (+1053874us) service_pool.cc:225] Handling call
0905 20:00:10.107611 (+ 25939us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:10.138910 (+ 31299us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:10.138939 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:10.188401 (+ 49462us) log.cc:844] Serialized 26 byte log entry
0905 20:00:10.596499 (+408098us) raft_consensus.cc:1643] Marking committed up to 3
0905 20:00:10.596547 (+ 48us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:10.596625 (+ 78us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:10.679417 (+ 82792us) raft_consensus.cc:1683] finished
0905 20:00:10.679471 (+ 54us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:10.679771 (+ 300us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3299,"raft.queue_time_us":9884,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":14578,"thread_start_us":367272,"threads_started":1,"wal-append.queue_time_us":457525}
W20250905 20:00:10.696393 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 235272us
W20250905 20:00:10.702667 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:10.662817 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:10.663378 30684 consensus_queue.cc:1035] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.066s
W20250905 20:00:10.729794 28122 leader_election.cc:336] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.649s (SENT)
W20250905 20:00:10.739754 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:10.736129 28076 raft_consensus.cc:1273] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250905 20:00:10.741701 28083 raft_consensus.cc:1273] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250905 20:00:10.694970 30756 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Advancing to term 8
W20250905 20:00:10.760591 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 73356.6us
I20250905 20:00:10.657193 30240 raft_consensus.cc:513] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 20:00:10.762286 28717 leader_election.cc:304] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:10.765646 28717 leader_election.cc:304] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:10.735145 30741 raft_consensus.cc:695] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 6 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:10.769248 28077 raft_consensus.cc:2391] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 5: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:10.764629 30690 raft_consensus.cc:2747] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:10.753435 30597 consensus_queue.cc:1046] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.019s
W20250905 20:00:10.783035 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 214364us
W20250905 20:00:10.785147 28122 leader_election.cc:336] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.697s (SENT)
I20250905 20:00:10.786734 30713 raft_consensus.cc:3053] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 5 LEADER]: Stepping down as leader of term 5
I20250905 20:00:10.787248 30713 raft_consensus.cc:738] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:10.788127 30713 consensus_queue.cc:260] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:10.789497 30713 raft_consensus.cc:3058] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:10.790449 30793 raft_consensus.cc:491] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:10.791003 30793 raft_consensus.cc:513] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:10.793272 30793 leader_election.cc:290] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:10.794675 28084 raft_consensus.cc:2466] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 20:00:10.795161 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:10.795778 30741 consensus_queue.cc:237] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:10.798539 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 339259us
I20250905 20:00:10.800953 28125 leader_election.cc:304] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:10.802701 30612 raft_consensus.cc:2802] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election won for term 6
I20250905 20:00:10.803462 30612 raft_consensus.cc:695] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:10.803500 28715 leader_election.cc:304] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:10.804673 30612 consensus_queue.cc:237] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:10.806667 30668 consensus_queue.cc:1046] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 20:00:10.808284 30668 raft_consensus.cc:3053] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 20:00:10.808795 30668 raft_consensus.cc:738] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:10.819624 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 20:00:10.825314 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 73081.2us
I20250905 20:00:10.831053 28125 leader_election.cc:304] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:10.809890 30668 consensus_queue.cc:260] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:10.837589 30594 raft_consensus.cc:2802] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 20:00:10.838152 30668 raft_consensus.cc:3058] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:10.838835 28125 leader_election.cc:304] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:10.838256 30594 raft_consensus.cc:695] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:10.840386 30537 raft_consensus.cc:2747] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250905 20:00:10.841204 30594 consensus_queue.cc:237] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:10.849783 30721 raft_consensus.cc:2802] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:10.810603 30823 consensus_queue.cc:1035] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.077s
I20250905 20:00:10.862253 30777 raft_consensus.cc:513] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:10.864311 30800 log.cc:927] Time spent T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.618s user 0.003s sys 0.000s
I20250905 20:00:10.865298 30777 leader_election.cc:290] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:10.866341 28079 raft_consensus.cc:1238] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:10.867626 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:10.868443 28152 raft_consensus.cc:3058] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:10.858016 30825 raft_consensus.cc:2802] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:10.870817 30825 raft_consensus.cc:491] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:10.871284 30825 raft_consensus.cc:3058] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Advancing to term 7
W20250905 20:00:10.871626 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 91610us
I20250905 20:00:10.873786 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20250905 20:00:10.876574 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:10.879627 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 45004.9us
I20250905 20:00:10.881469 30668 raft_consensus.cc:491] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:10.882045 30668 raft_consensus.cc:513] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:10.882817 30775 raft_consensus.cc:513] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:10.884214 30668 leader_election.cc:290] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 20:00:10.863207 30814 log.cc:927] Time spent T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20250905 20:00:10.886216 28085 raft_consensus.cc:2466] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
W20250905 20:00:10.891734 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:10.894671 28122 leader_election.cc:336] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.817s (SENT)
I20250905 20:00:10.900605 28122 leader_election.cc:304] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
W20250905 20:00:10.901038 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:10.902177 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 136606us
I20250905 20:00:10.850306 30721 raft_consensus.cc:491] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:10.884752 30775 leader_election.cc:290] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:10.904318 30240 leader_election.cc:290] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 20:00:10.937548 30799 log.cc:927] Time spent T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.855s user 0.005s sys 0.002s
I20250905 20:00:10.936872 30588 consensus_queue.cc:1046] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20250905 20:00:10.943861 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:10.949817 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:10.950372 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:10.906126 30721 raft_consensus.cc:3058] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:10.950539 30672 raft_consensus.cc:513] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:10.954102 30672 leader_election.cc:290] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:10.949599 30588 consensus_queue.cc:1035] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.001s
I20250905 20:00:10.949762 30684 raft_consensus.cc:3053] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Stepping down as leader of term 4
I20250905 20:00:10.944936 27973 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:10.955690 30684 raft_consensus.cc:738] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
W20250905 20:00:10.956698 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:10.902858 30594 raft_consensus.cc:2747] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:10.956854 30684 consensus_queue.cc:260] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:10.957466 30756 raft_consensus.cc:2747] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250905 20:00:10.958060 30588 consensus_queue.cc:1035] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
W20250905 20:00:10.961826 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:10.961720 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:10.962870 28157 raft_consensus.cc:3058] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
W20250905 20:00:10.966914 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 72324.2us
I20250905 20:00:10.937153 27975 leader_election.cc:400] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705) with higher term. Message: Invalid argument: T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 3. Current term is 4.
W20250905 20:00:10.918773 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 115363us
I20250905 20:00:10.958300 30684 raft_consensus.cc:3058] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
W20250905 20:00:10.971978 28084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1514) took 1941 ms (client timeout 1577 ms). Trace:
W20250905 20:00:10.972432 28084 rpcz_store.cc:269] 0905 20:00:09.035364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:09.055221 (+ 19857us) service_pool.cc:225] Handling call
0905 20:00:10.971730 (+1916509us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54000f3f54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:10.971885 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2886090624}
I20250905 20:00:10.974860 28159 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39586 (request call id 316) took 1048 ms. Trace:
I20250905 20:00:10.975335 28159 rpcz_store.cc:276] 0905 20:00:09.932368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:09.943127 (+ 10759us) service_pool.cc:225] Handling call
0905 20:00:10.974680 (+1031553us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b5400073ad4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:10.974802 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":349123072}
I20250905 20:00:10.903036 30831 raft_consensus.cc:2747] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250905 20:00:10.977300 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250905 20:00:10.988116 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:10.987171 28152 raft_consensus.cc:2466] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
W20250905 20:00:10.989883 28007 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1478) took 4221 ms (client timeout 2636 ms). Trace:
I20250905 20:00:10.990231 28050 leader_election.cc:304] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 20:00:10.990401 28007 rpcz_store.cc:269] 0905 20:00:06.760269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:06.760455 (+ 186us) service_pool.cc:225] Handling call
0905 20:00:10.967794 (+4207339us) spinlock_profiling.cc:243] Waited 3.35 s on lock 0x7b54001fc054. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:10.978554 (+ 10760us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7714097536}
I20250905 20:00:10.658362 28080 rpcz_store.cc:276] 0905 20:00:08.809428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:09.780112 (+970684us) service_pool.cc:225] Handling call
0905 20:00:09.938554 (+158442us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:09.965030 (+ 26476us) raft_consensus.cc:1532] Early marking committed up to index 5
0905 20:00:10.188527 (+223497us) log.cc:844] Serialized 14 byte log entry
0905 20:00:10.645216 (+456689us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:10.645337 (+ 121us) raft_consensus.cc:1643] Marking committed up to 5
0905 20:00:10.645427 (+ 90us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:10.645564 (+ 137us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:10.645831 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9901,"prepare.queue_time_us":430004,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":2662,"raft.queue_time_us":18025,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":7699,"thread_start_us":544957,"threads_started":2}
I20250905 20:00:10.994405 30672 raft_consensus.cc:2802] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election won for term 5
I20250905 20:00:10.995100 30672 raft_consensus.cc:695] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [term 5 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:10.996471 30672 consensus_queue.cc:237] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:10.999888 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20250905 20:00:11.000487 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 63598.1us
I20250905 20:00:10.976692 30663 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250905 20:00:11.002575 30807 log.cc:927] Time spent T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.340s user 0.000s sys 0.009s
I20250905 20:00:11.002781 30663 raft_consensus.cc:3058] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:11.004074 28745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40926 (request call id 495) took 1031 ms. Trace:
I20250905 20:00:10.902254 30148 consensus_queue.cc:1035] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.086s
I20250905 20:00:11.004447 28745 rpcz_store.cc:276] 0905 20:00:09.972929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:09.979435 (+ 6506us) service_pool.cc:225] Handling call
0905 20:00:09.989259 (+ 9824us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:09.989812 (+ 553us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:09.989846 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:09.990394 (+ 548us) log.cc:844] Serialized 26 byte log entry
0905 20:00:10.443882 (+453488us) raft_consensus.cc:1643] Marking committed up to 6
0905 20:00:10.443962 (+ 80us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:10.444065 (+ 103us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:10.943559 (+499494us) spinlock_profiling.cc:243] Waited 4.82 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49d85891e 00007fe49d857c66 00007fe49d857888 00007fe49d8612c1 00007fe49d861279 00007fe49d861209
0905 20:00:11.003773 (+ 60214us) raft_consensus.cc:1683] finished
0905 20:00:11.003793 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:11.004013 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2660,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":88,"spinlock_wait_cycles":11090688,"thread_start_us":452119,"threads_started":1,"wal-append.queue_time_us":592600}
W20250905 20:00:10.974965 30816 log.cc:927] Time spent T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.183s user 0.000s sys 0.002s
I20250905 20:00:10.939363 30559 leader_election.cc:290] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:11.044523 28157 raft_consensus.cc:2466] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 6.
I20250905 20:00:11.046116 28050 leader_election.cc:304] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 20:00:11.057083 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 218390us
I20250905 20:00:11.035108 30660 raft_consensus.cc:491] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:11.063336 30670 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:11.063859 30635 raft_consensus.cc:491] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:11.103832 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 128711us
I20250905 20:00:11.104828 28125 leader_election.cc:304] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 20:00:11.064780 27907 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34448 (request call id 254) took 14071 ms (client timeout 15000 ms). Trace:
I20250905 20:00:11.065444 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:11.065466 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:11.066113 30812 log.cc:927] Time spent T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.403s user 0.001s sys 0.002s
I20250905 20:00:11.069597 27904 catalog_manager.cc:5582] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 3 to 5, leader changed from 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 5 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 20:00:11.070955 30675 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:11.070847 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:11.108984 28077 raft_consensus.cc:2391] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 5: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:11.071719 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:11.071727 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:11.073529 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:11.074208 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:11.075330 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:11.077608 30829 log.cc:927] Time spent T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.078s user 0.001s sys 0.001s
I20250905 20:00:11.082865 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:11.083710 28748 raft_consensus.cc:1238] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 6 LEADER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:11.085168 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:11.086822 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:11.093387 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 56559.8us
I20250905 20:00:11.093778 30823 raft_consensus.cc:3053] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Stepping down as leader of term 4
I20250905 20:00:11.032933 30831 consensus_queue.cc:1046] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20250905 20:00:11.095302 28001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1486) took 1017 ms. Trace:
I20250905 20:00:11.018301 30756 consensus_queue.cc:1035] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
W20250905 20:00:11.100445 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:11.114507 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:11.050843 30777 raft_consensus.cc:2802] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election won for term 6
I20250905 20:00:11.114574 28082 raft_consensus.cc:2466] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6.
W20250905 20:00:11.126536 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:11.125479 28152 raft_consensus.cc:1238] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 3. Current term is 4. Ops: []
I20250905 20:00:11.110809 28081 raft_consensus.cc:2466] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6.
I20250905 20:00:11.131398 28080 raft_consensus.cc:2466] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 20:00:11.132418 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:11.133347 28076 raft_consensus.cc:2391] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 6: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:11.134831 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:11.136981 27904 catalog_manager.cc:5582] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 20:00:11.137692 28157 raft_consensus.cc:1238] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:11.127946 30823 raft_consensus.cc:738] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:11.141549 30823 consensus_queue.cc:260] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:11.143297 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:11.143632 30823 raft_consensus.cc:3058] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:11.144807 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:11.145493 28080 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 20:00:11.146801 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:11.105974 27907 rpcz_store.cc:269] 0905 19:59:56.993207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:56.995987 (+ 2780us) service_pool.cc:225] Handling call
0905 19:59:57.858719 (+862732us) spinlock_profiling.cc:243] Waited 436 ms on lock 0x7b54001bec54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9dcd0 00007fe49dc21eea 00007fe4a394718c 00007fe4a3a3fb6e 00007fe4a1f88484 00007fe4a1f88412 00007fe4a1f88341 00007fe4a1f882bc 00007fe4a1f87572 00007fe49c018ddc 00007fe49c018216
0905 20:00:11.064549 (+13205830us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 20:00:02.352280 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 20:00:02.368126 (+ 15846us) write_op.cc:432] Acquiring schema lock in shared mode
0905 20:00:02.368184 (+ 58us) write_op.cc:435] Acquired schema lock
0905 20:00:02.368206 (+ 22us) tablet.cc:660] Decoding operations
0905 20:00:02.450359 (+ 82153us) write_op.cc:620] Acquiring the partition lock for write op
0905 20:00:02.450492 (+ 133us) write_op.cc:641] Partition lock acquired for write op
0905 20:00:02.450539 (+ 47us) tablet.cc:683] Acquiring locks for 16 operations
0905 20:00:02.451911 (+ 1372us) tablet.cc:699] Row locks acquired
0905 20:00:02.451926 (+ 15us) write_op.cc:260] PREPARE: finished
0905 20:00:02.452118 (+ 192us) write_op.cc:270] Start()
0905 20:00:02.452296 (+ 178us) write_op.cc:276] Timestamp: P: 1757102402452044 usec, L: 0
0905 20:00:02.452313 (+ 17us) op_driver.cc:348] REPLICATION: starting
0905 20:00:02.453178 (+ 865us) log.cc:844] Serialized 6400 byte log entry
0905 20:00:04.409568 (+1956390us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49d85891e 00007fe49d857c66 00007fe49d857888 00007fe49d8612c1 00007fe49d861279 00007fe49d861209
0905 20:00:05.536341 (+1126773us) op_driver.cc:464] REPLICATION: finished
0905 20:00:05.904065 (+367724us) write_op.cc:301] APPLY: starting
0905 20:00:05.913236 (+ 9171us) tablet.cc:1365] starting BulkCheckPresence
0905 20:00:06.022160 (+108924us) tablet.cc:1368] finished BulkCheckPresence
0905 20:00:06.030991 (+ 8831us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 20:00:06.409659 (+378668us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 20:00:06.554908 (+145249us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=16
0905 20:00:06.582785 (+ 27877us) write_op.cc:312] APPLY: finished
0905 20:00:06.869222 (+286437us) log.cc:844] Serialized 113 byte log entry
0905 20:00:09.483548 (+2614326us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49ac86fec 00007fe4a38dcb99 00007fe49aca369b 00007fe49aca193f 00007fe49aca3c02 00007fe49d8572bb
0905 20:00:09.986637 (+503089us) write_op.cc:489] Releasing partition, row and schema locks
0905 20:00:09.986927 (+ 290us) write_op.cc:454] Released schema lock
0905 20:00:09.987073 (+ 146us) write_op.cc:341] FINISH: Updating metrics
0905 20:00:09.987738 (+ 665us) write_op.cc:489] Releasing partition, row and schema locks
0905 20:00:09.987781 (+ 43us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":37,"spinlock_wait_cycles":1001652992,"child_traces":[["op",{"apply.queue_time_us":340748,"apply.run_cpu_time_us":43806,"apply.run_wall_time_us":4092156,"num_ops":16,"prepare.queue_time_us":213540,"prepare.run_cpu_time_us":22138,"prepare.run_wall_time_us":176685,"raft.queue_time_us":595927,"raft.run_cpu_time_us":30961,"raft.run_wall_time_us":380487,"replication_time_us":2986517,"spinlock_wait_cycles":58056320,"thread_start_us":3826898,"threads_started":5,"wal-append.queue_time_us":3298230,"wal-append.run_cpu_time_us":59068,"wal-append.run_wall_time_us":3692944}]]}
I20250905 20:00:11.152136 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:11.155016 30597 raft_consensus.cc:3053] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 LEADER]: Stepping down as leader of term 3
I20250905 20:00:11.155597 30597 raft_consensus.cc:738] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
W20250905 20:00:11.155616 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:11.156487 30597 consensus_queue.cc:260] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:11.157840 30597 raft_consensus.cc:3058] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Advancing to term 5
I20250905 20:00:11.158165 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:11.158757 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:11.161140 27973 leader_election.cc:304] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 20:00:11.161548 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 56858.6us
I20250905 20:00:11.161489 28085 raft_consensus.cc:2466] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6.
I20250905 20:00:11.162627 30835 raft_consensus.cc:2747] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250905 20:00:11.122866 30675 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:11.164470 30775 consensus_queue.cc:1035] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 20:00:11.122406 30635 raft_consensus.cc:513] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:11.166005 30775 raft_consensus.cc:2802] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:11.167197 30635 leader_election.cc:290] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:11.167773 30612 raft_consensus.cc:491] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:11.168018 28715 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:11.168376 30612 raft_consensus.cc:513] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:11.169103 30741 raft_consensus.cc:2747] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:11.169391 28715 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:11.170255 30824 raft_consensus.cc:2747] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250905 20:00:11.109377 30660 raft_consensus.cc:513] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:11.170928 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:11.170929 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:11.035323 30721 raft_consensus.cc:513] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:11.173895 30721 leader_election.cc:290] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:11.175076 30663 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:11.176188 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:11.176606 28150 raft_consensus.cc:1273] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250905 20:00:11.176805 28083 raft_consensus.cc:2466] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:11.175338 30756 consensus_queue.cc:1046] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250905 20:00:11.177217 28084 raft_consensus.cc:2466] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 20:00:11.177419 30663 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:11.178332 28158 raft_consensus.cc:1238] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:11.175545 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:11.180995 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:11.182650 30777 raft_consensus.cc:695] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:11.175571 30825 raft_consensus.cc:513] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:11.183130 28158 raft_consensus.cc:1238] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:11.183635 30777 consensus_queue.cc:237] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:11.184284 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:11.184662 30825 leader_election.cc:290] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:11.184974 28150 raft_consensus.cc:2466] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:11.185966 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:11.128505 30670 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:11.164749 30675 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:11.188735 28715 leader_election.cc:304] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:11.140872 27904 catalog_manager.cc:5582] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 2 to 3, leader changed from cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 3 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20250905 20:00:11.190482 28077 raft_consensus.cc:2466] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:11.202890 28715 leader_election.cc:304] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:11.221411 28715 leader_election.cc:304] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:11.225101 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:11.225795 28084 raft_consensus.cc:2466] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 20:00:11.227712 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:11.228521 28154 raft_consensus.cc:2391] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 4: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:11.188158 30775 raft_consensus.cc:491] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:11.126782 28001 rpcz_store.cc:276] 0905 20:00:10.078126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:10.091307 (+ 13181us) service_pool.cc:225] Handling call
0905 20:00:10.114670 (+ 23363us) raft_consensus.cc:1469] Updating replica for 2 ops
0905 20:00:10.137067 (+ 22397us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:10.137284 (+ 217us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0905 20:00:10.160568 (+ 23284us) log.cc:844] Serialized 52 byte log entry
0905 20:00:10.627221 (+466653us) raft_consensus.cc:1643] Marking committed up to 3
0905 20:00:10.760234 (+133013us) log.cc:844] Serialized 14 byte log entry
0905 20:00:10.760483 (+ 249us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:10.760576 (+ 93us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:10.974275 (+213699us) spinlock_profiling.cc:243] Waited 7.67 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49d85891e 00007fe49d857c66 00007fe49d857888 00007fe49d8612c1 00007fe49d861279 00007fe49d861209
0905 20:00:11.094737 (+120462us) raft_consensus.cc:1683] finished
0905 20:00:11.094928 (+ 191us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:11.095106 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7122,"prepare.queue_time_us":194757,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":22618,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":5871,"spinlock_wait_cycles":17634560,"thread_start_us":511890,"threads_started":2,"wal-append.queue_time_us":604083}
I20250905 20:00:11.203348 28076 raft_consensus.cc:2466] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 20:00:11.210119 28152 raft_consensus.cc:2466] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
W20250905 20:00:11.236516 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 48400.1us
I20250905 20:00:11.200843 30588 raft_consensus.cc:3053] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Stepping down as leader of term 5
W20250905 20:00:11.238134 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 119520us
W20250905 20:00:11.242544 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:11.204180 30660 leader_election.cc:290] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:11.220824 30825 raft_consensus.cc:2747] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 20:00:11.251361 28159 raft_consensus.cc:1238] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 6. Current term is 7. Ops: []
W20250905 20:00:11.251999 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:11.224690 30721 raft_consensus.cc:2802] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:11.236940 30775 raft_consensus.cc:3058] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:11.253420 28715 leader_election.cc:304] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:11.255749 28122 leader_election.cc:304] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:11.196522 30756 consensus_queue.cc:1035] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.051s
I20250905 20:00:11.226802 30612 leader_election.cc:290] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:11.251068 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:11.247977 30588 raft_consensus.cc:738] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:11.268966 28748 raft_consensus.cc:3058] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:11.254348 30824 raft_consensus.cc:2802] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:11.279932 27904 catalog_manager.cc:5582] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 4 to 6, leader changed from 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 6 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20250905 20:00:11.259990 30721 raft_consensus.cc:491] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:11.297096 27904 catalog_manager.cc:5582] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 1 to 2, leader changed from 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 2 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20250905 20:00:11.297204 28152 raft_consensus.cc:1273] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20250905 20:00:11.300192 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:11.303301 28715 leader_election.cc:304] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 20:00:11.304900 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 52941.6us
I20250905 20:00:11.258184 30263 consensus_queue.cc:1035] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.038s
I20250905 20:00:11.301412 28152 pending_rounds.cc:85] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting all ops after (but not including) 2
I20250905 20:00:11.308727 28152 pending_rounds.cc:107] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20250905 20:00:11.309016 28152 raft_consensus.cc:2887] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250905 20:00:11.309535 27904 catalog_manager.cc:5582] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20250905 20:00:11.302659 30721 raft_consensus.cc:3058] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:11.318667 28125 leader_election.cc:304] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:11.319025 30670 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:11.300323 30612 raft_consensus.cc:2747] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:11.300772 30825 raft_consensus.cc:2802] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250905 20:00:11.322501 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 70189.4us
I20250905 20:00:11.261207 30840 consensus_queue.cc:1035] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20250905 20:00:11.327013 28748 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:11.307976 30741 raft_consensus.cc:2802] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:11.329823 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:11.332027 28123 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
W20250905 20:00:11.333235 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:11.336267 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:11.333993 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 20:00:11.339251 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:11.340524 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 11 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:11.341970 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:11.351101 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:11.351775 28155 raft_consensus.cc:3058] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Advancing to term 7
W20250905 20:00:11.355260 30842 log.cc:927] Time spent T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20250905 20:00:11.357280 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:11.325512 30825 raft_consensus.cc:491] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:11.364809 28717 leader_election.cc:304] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:11.320259 30240 consensus_queue.cc:1046] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20250905 20:00:11.332139 30741 raft_consensus.cc:491] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:11.337013 30663 raft_consensus.cc:2802] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election won for term 5
I20250905 20:00:11.371781 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:11.394815 28159 raft_consensus.cc:2391] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 6: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
W20250905 20:00:11.371243 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 43979.3us
I20250905 20:00:11.397231 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:11.397878 28001 raft_consensus.cc:3058] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
W20250905 20:00:11.457162 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 57982.3us
I20250905 20:00:11.331712 30668 consensus_queue.cc:1046] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.066s
I20250905 20:00:11.307356 30588 consensus_queue.cc:260] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:11.462030 28157 raft_consensus.cc:1238] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:11.371335 30825 raft_consensus.cc:3058] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Advancing to term 6
W20250905 20:00:11.473071 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 101889us
I20250905 20:00:11.477254 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:11.478963 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:11.481113 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:11.483325 28005 raft_consensus.cc:1238] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:11.485010 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:11.486171 28003 raft_consensus.cc:2391] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:11.366959 30756 consensus_queue.cc:1046] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.096s
I20250905 20:00:11.394059 30741 raft_consensus.cc:3058] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:11.395869 30830 raft_consensus.cc:2747] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 20:00:11.393982 30612 raft_consensus.cc:2802] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election won for term 6
W20250905 20:00:11.468753 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:11.456034 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 20:00:11.486862 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 88945.1us
I20250905 20:00:11.513587 28010 raft_consensus.cc:1238] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: []
I20250905 20:00:11.494457 28122 leader_election.cc:304] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 20:00:11.515903 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 186325us
W20250905 20:00:11.521204 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 241670us
I20250905 20:00:11.522369 27973 leader_election.cc:304] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:11.528811 28084 raft_consensus.cc:1273] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20250905 20:00:11.530105 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:11.533339 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:11.514253 30675 raft_consensus.cc:491] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
W20250905 20:00:11.559811 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 97035.4us
I20250905 20:00:11.542742 30612 raft_consensus.cc:695] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:11.564868 30240 raft_consensus.cc:491] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:11.596717 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:11.589919 28122 leader_election.cc:304] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:11.541451 30831 consensus_queue.cc:1035] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20250905 20:00:11.592842 28003 raft_consensus.cc:1238] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250905 20:00:11.606736 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:11.608318 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:11.611809 28748 raft_consensus.cc:1238] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 3. Current term is 4. Ops: []
I20250905 20:00:11.614055 28005 raft_consensus.cc:1238] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:11.618139 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:11.620390 28009 raft_consensus.cc:1238] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 7. Current term is 8. Ops: []
I20250905 20:00:11.623912 28006 raft_consensus.cc:1238] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250905 20:00:11.579948 30861 raft_consensus.cc:2802] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 20:00:11.627983 28004 raft_consensus.cc:3058] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:11.629806 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:11.629051 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:11.632246 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:11.633333 28008 raft_consensus.cc:2391] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:11.572582 30675 raft_consensus.cc:513] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:11.605472 30588 raft_consensus.cc:3058] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:11.640372 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:11.640703 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:11.639561 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 108434us
W20250905 20:00:11.631515 30854 log.cc:927] Time spent T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.163s user 0.000s sys 0.001s
I20250905 20:00:11.652639 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:11.642135 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 20:00:11.659768 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:11.666007 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:11.667968 30859 raft_consensus.cc:3053] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Stepping down as leader of term 4
I20250905 20:00:11.668639 30859 raft_consensus.cc:738] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:11.672243 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:11.615931 30240 raft_consensus.cc:513] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:11.638535 28744 raft_consensus.cc:1238] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 8. Current term is 9. Ops: []
I20250905 20:00:11.678648 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:11.678947 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:11.682757 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:11.640760 30866 raft_consensus.cc:2802] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader election won for term 8
I20250905 20:00:11.690385 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:11.642238 30663 raft_consensus.cc:695] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:11.696002 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:11.696993 28008 raft_consensus.cc:2466] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
W20250905 20:00:11.682392 30845 log.cc:927] Time spent T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.294s user 0.000s sys 0.002s
W20250905 20:00:11.704185 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:11.713745 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:11.715858 28718 leader_election.cc:304] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:11.717576 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:11.718847 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:11.720671 28006 raft_consensus.cc:2408] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
W20250905 20:00:11.723079 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 82983.5us
I20250905 20:00:11.689143 30860 raft_consensus.cc:2747] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250905 20:00:11.728338 28751 raft_consensus.cc:1238] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:11.669847 30859 consensus_queue.cc:260] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:11.712564 30851 log.cc:927] Time spent T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.307s user 0.000s sys 0.002s
I20250905 20:00:11.731823 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 20:00:11.732331 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:11.732419 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:11.728300 30672 raft_consensus.cc:491] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:11.733457 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:11.778381 28005 raft_consensus.cc:2391] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:11.733614 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:11.779704 28077 raft_consensus.cc:2366] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6.
I20250905 20:00:11.734790 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:11.736368 30713 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:11.728572 30756 consensus_queue.cc:1046] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20250905 20:00:11.726789 30832 raft_consensus.cc:3058] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:11.739657 30660 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
W20250905 20:00:11.743362 30868 log.cc:927] Time spent T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
W20250905 20:00:11.743758 30838 log.cc:927] Time spent T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.584s user 0.005s sys 0.000s
W20250905 20:00:11.759953 30864 log.cc:927] Time spent T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.182s user 0.000s sys 0.001s
I20250905 20:00:11.769685 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:11.776386 28155 raft_consensus.cc:2466] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 7.
W20250905 20:00:11.781903 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:11.784606 28718 leader_election.cc:304] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:11.788029 30672 raft_consensus.cc:513] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:11.713168 30670 raft_consensus.cc:491] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:11.786362 28083 raft_consensus.cc:2391] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 4: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:11.722539 30866 raft_consensus.cc:695] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:11.839077 30823 raft_consensus.cc:2747] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250905 20:00:11.839696 30866 consensus_queue.cc:237] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:11.830529 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:11.842602 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:11.843324 28006 raft_consensus.cc:3058] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:11.843531 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:11.844964 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:11.830940 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 50252us
I20250905 20:00:11.830504 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:11.838177 30670 raft_consensus.cc:513] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:11.848461 28005 raft_consensus.cc:2408] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20250905 20:00:11.846179 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:11.856940 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:11.862380 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 77884us
I20250905 20:00:11.863860 28718 leader_election.cc:304] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:11.743367 30148 consensus_queue.cc:1046] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.105s
W20250905 20:00:11.872769 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 81808.3us
I20250905 20:00:11.703130 30612 consensus_queue.cc:237] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:11.877203 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:11.874311 30831 raft_consensus.cc:2747] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:11.884294 30672 leader_election.cc:290] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:11.793092 30713 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:11.886832 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:11.918741 28005 raft_consensus.cc:2391] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 9: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:11.907732 28718 leader_election.cc:304] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:11.849776 30846 raft_consensus.cc:491] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:11.896277 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 20:00:11.933115 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 60349.7us
I20250905 20:00:11.863813 30859 raft_consensus.cc:3058] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:11.918880 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:11.886125 30844 raft_consensus.cc:3053] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Stepping down as leader of term 5
I20250905 20:00:11.938679 30844 raft_consensus.cc:738] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:11.938956 28125 leader_election.cc:304] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 20:00:11.940163 30597 consensus_meta.cc:287] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.782s user 0.011s sys 0.006s
I20250905 20:00:11.940447 28125 leader_election.cc:304] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:11.940912 28745 raft_consensus.cc:2466] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
W20250905 20:00:11.942162 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 68344.5us
I20250905 20:00:11.932762 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:11.937388 30668 consensus_queue.cc:1046] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250905 20:00:11.937987 30240 leader_election.cc:290] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:11.939839 30867 consensus_queue.cc:1046] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250905 20:00:11.892961 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:11.942047 28125 leader_election.cc:304] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:11.942912 28122 leader_election.cc:304] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:11.886039 30670 leader_election.cc:290] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:11.951962 28081 raft_consensus.cc:2391] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 4: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
W20250905 20:00:11.945142 28150 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.016s sys 0.028s
I20250905 20:00:11.950603 27973 leader_election.cc:400] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Vote denied by peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193) with higher term. Message: Invalid argument: T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6.
I20250905 20:00:11.953434 27973 leader_election.cc:403] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20250905 20:00:11.946473 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:11.836329 30660 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:11.830687 30860 consensus_queue.cc:1046] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.090s
I20250905 20:00:11.960445 28008 raft_consensus.cc:1238] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:11.891420 30537 raft_consensus.cc:3053] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Stepping down as leader of term 4
W20250905 20:00:11.955389 28150 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.583s user 0.016s sys 0.028s
I20250905 20:00:11.947793 30834 raft_consensus.cc:2747] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250905 20:00:11.964752 30537 raft_consensus.cc:738] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:11.965088 30713 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:11.965894 30537 consensus_queue.cc:260] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:11.967123 30537 raft_consensus.cc:3058] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:11.852223 30663 consensus_queue.cc:237] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:11.921227 30675 leader_election.cc:290] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:11.978590 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:11.985841 28158 raft_consensus.cc:2466] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 6.
W20250905 20:00:11.991727 28001 consensus_meta.cc:287] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.528s user 0.020s sys 0.001s
I20250905 20:00:11.992606 28001 raft_consensus.cc:2466] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
W20250905 20:00:11.994578 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 52590.2us
I20250905 20:00:11.996491 28125 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:12.021780 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:12.001773 28009 raft_consensus.cc:2466] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6.
I20250905 20:00:12.022715 28005 raft_consensus.cc:2391] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:12.030905 30877 raft_consensus.cc:491] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:12.031594 30877 raft_consensus.cc:513] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:12.033838 30877 leader_election.cc:290] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:12.034761 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 20:00:12.035202 30775 consensus_meta.cc:287] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.749s user 0.019s sys 0.001s
I20250905 20:00:12.036280 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 20:00:11.998328 30825 consensus_meta.cc:287] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.526s user 0.009s sys 0.009s
I20250905 20:00:12.013857 30756 raft_consensus.cc:3058] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:11.946835 30846 raft_consensus.cc:513] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:12.035768 30775 raft_consensus.cc:513] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:12.047713 28158 raft_consensus.cc:1273] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250905 20:00:12.048403 30775 leader_election.cc:290] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:12.049247 30876 log.cc:927] Time spent T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.113s user 0.000s sys 0.000s
I20250905 20:00:12.051160 28007 raft_consensus.cc:2366] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 5. Current term is 6.
I20250905 20:00:12.052882 28004 raft_consensus.cc:2466] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 20:00:12.037415 30875 raft_consensus.cc:3058] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:12.037549 30869 raft_consensus.cc:2747] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250905 20:00:12.037065 30825 raft_consensus.cc:513] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:12.046418 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:11.963217 30670 raft_consensus.cc:491] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:12.037561 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 20:00:12.054133 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 103697us
W20250905 20:00:12.039759 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 96910.1us
I20250905 20:00:12.070395 27973 leader_election.cc:304] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:12.070760 28122 leader_election.cc:304] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:12.072059 30775 raft_consensus.cc:2747] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 20:00:12.072402 28122 leader_election.cc:304] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 20:00:12.072906 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:12.045442 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:12.074467 28155 raft_consensus.cc:2391] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 4: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:12.037153 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:12.075471 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:12.076450 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:12.077230 28008 raft_consensus.cc:2466] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
I20250905 20:00:12.045177 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:12.043256 28076 raft_consensus.cc:1273] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250905 20:00:12.078284 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:12.043820 28006 raft_consensus.cc:2466] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
I20250905 20:00:12.066484 28081 raft_consensus.cc:1273] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250905 20:00:12.067334 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:12.081653 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:12.084618 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:12.085280 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:12.053865 30861 raft_consensus.cc:491] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:12.091326 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:12.092288 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:12.092410 28751 raft_consensus.cc:1238] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:12.044485 30240 raft_consensus.cc:2747] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:12.096565 28007 raft_consensus.cc:2391] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:12.096612 28008 raft_consensus.cc:2408] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20250905 20:00:12.100922 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 104543us
I20250905 20:00:12.101387 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:12.103901 28001 raft_consensus.cc:2408] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20250905 20:00:12.104693 28747 raft_consensus.cc:1238] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6. Ops: []
W20250905 20:00:12.105625 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:12.106752 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:12.107911 28006 raft_consensus.cc:2366] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 5. Current term is 6.
I20250905 20:00:12.108816 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:12.109109 28076 raft_consensus.cc:1273] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250905 20:00:12.111092 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:12.111992 28003 raft_consensus.cc:2391] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:12.069245 30713 raft_consensus.cc:491] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:12.113454 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:12.105219 28125 leader_election.cc:304] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:12.114560 28006 raft_consensus.cc:2374] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4. Re-sending same reply.
W20250905 20:00:12.119076 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:12.119684 28076 raft_consensus.cc:1273] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250905 20:00:12.119984 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:12.122676 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:12.104219 30675 raft_consensus.cc:491] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:12.135797 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:12.145882 28005 raft_consensus.cc:2391] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 6: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
W20250905 20:00:12.136152 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:12.096316 30861 raft_consensus.cc:3058] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:12.107964 30832 raft_consensus.cc:2747] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:12.136790 28001 raft_consensus.cc:2391] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 4: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:12.112953 30869 raft_consensus.cc:2802] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:12.143563 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:12.152561 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 47441.5us
I20250905 20:00:12.152598 28003 raft_consensus.cc:2391] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:12.148496 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:12.121279 30721 consensus_meta.cc:287] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.804s user 0.013s sys 0.000s
I20250905 20:00:12.089040 30670 raft_consensus.cc:513] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:12.155478 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 83169.2us
I20250905 20:00:12.118559 30867 raft_consensus.cc:2802] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 20:00:12.142396 30240 raft_consensus.cc:2747] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250905 20:00:12.163080 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:12.166827 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:12.013070 30844 consensus_queue.cc:260] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:12.161485 28125 leader_election.cc:304] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:12.165297 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 11 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:12.170852 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:12.150779 30672 raft_consensus.cc:491] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:12.196295 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:12.193202 28717 leader_election.cc:304] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:12.197350 28001 raft_consensus.cc:2408] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20250905 20:00:12.154758 30869 raft_consensus.cc:491] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:12.121304 30713 raft_consensus.cc:513] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:12.202258 28155 raft_consensus.cc:1273] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20250905 20:00:12.204020 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 20:00:12.213001 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:12.211532 28153 raft_consensus.cc:1273] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20250905 20:00:12.211798 28005 raft_consensus.cc:1238] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 8. Current term is 9. Ops: []
I20250905 20:00:12.216323 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:12.215250 28153 pending_rounds.cc:85] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting all ops after (but not including) 2
W20250905 20:00:12.214680 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 160931us
I20250905 20:00:12.218237 28153 pending_rounds.cc:107] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20250905 20:00:12.218621 28153 raft_consensus.cc:2887] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250905 20:00:12.219331 28158 raft_consensus.cc:1273] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250905 20:00:12.220811 30588 consensus_queue.cc:1046] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250905 20:00:12.221668 30831 consensus_queue.cc:1035] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250905 20:00:12.168608 30867 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:12.184357 30537 raft_consensus.cc:2747] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250905 20:00:12.222694 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:12.212570 28155 pending_rounds.cc:85] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting all ops after (but not including) 2
I20250905 20:00:12.233731 28155 pending_rounds.cc:107] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20250905 20:00:12.234102 28155 raft_consensus.cc:2887] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250905 20:00:12.227603 28007 raft_consensus.cc:1273] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250905 20:00:12.183566 30891 raft_consensus.cc:2802] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 20:00:12.196445 30773 raft_consensus.cc:491] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:12.230877 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:12.152413 30675 raft_consensus.cc:513] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:12.208051 30869 raft_consensus.cc:3058] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:12.163467 30721 raft_consensus.cc:513] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:12.226198 30894 raft_consensus.cc:3058] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:12.225997 30823 raft_consensus.cc:2802] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader election won for term 7
W20250905 20:00:12.357640 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:12.268169 28004 raft_consensus.cc:1238] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:12.273144 28125 leader_election.cc:304] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:12.228927 30588 consensus_queue.cc:1046] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.032s
W20250905 20:00:12.360296 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 87239.4us
W20250905 20:00:12.360713 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 141087us
I20250905 20:00:12.361403 27975 leader_election.cc:304] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
W20250905 20:00:12.361581 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 100739us
W20250905 20:00:12.286863 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:12.291850 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:12.369145 28750 raft_consensus.cc:1238] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 6. Current term is 7. Ops: []
W20250905 20:00:12.372315 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 77746.6us
W20250905 20:00:12.263912 30878 log.cc:927] Time spent T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.234s user 0.002s sys 0.000s
W20250905 20:00:12.253023 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:12.243808 28156 raft_consensus.cc:1273] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20250905 20:00:12.254655 28051 leader_election.cc:304] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:12.228849 30867 raft_consensus.cc:3058] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:12.227974 30879 raft_consensus.cc:491] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:12.226486 30846 leader_election.cc:290] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:12.204090 30672 raft_consensus.cc:513] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:12.266093 30824 raft_consensus.cc:491] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:12.383073 28156 pending_rounds.cc:85] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting all ops after (but not including) 3
I20250905 20:00:12.352950 30884 raft_consensus.cc:491] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:12.394691 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:12.265815 30594 raft_consensus.cc:3053] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Stepping down as leader of term 4
I20250905 20:00:12.352965 30263 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:12.396771 28156 pending_rounds.cc:107] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20250905 20:00:12.405887 28156 raft_consensus.cc:2887] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250905 20:00:12.284967 30844 raft_consensus.cc:3058] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:12.373534 30835 raft_consensus.cc:2802] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 20:00:12.372608 30825 leader_election.cc:290] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:12.388291 30823 raft_consensus.cc:695] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 7 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:12.378403 30871 raft_consensus.cc:2747] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250905 20:00:12.360984 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:12.272294 30240 consensus_queue.cc:1035] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.124s
I20250905 20:00:12.407331 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:12.407675 28718 leader_election.cc:304] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:12.259889 30897 consensus_queue.cc:1046] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.163s
I20250905 20:00:12.398661 30824 raft_consensus.cc:3058] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:12.372789 30850 consensus_queue.cc:1046] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.288s
I20250905 20:00:12.380589 30775 consensus_queue.cc:1046] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.031s
I20250905 20:00:12.403723 30660 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:12.403546 30594 raft_consensus.cc:738] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:12.446964 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:12.441552 28049 leader_election.cc:304] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:12.418805 30891 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250905 20:00:12.453070 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 45512.1us
I20250905 20:00:12.256613 30773 raft_consensus.cc:513] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 20:00:12.429101 30835 raft_consensus.cc:695] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:12.442317 30832 raft_consensus.cc:2802] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:12.414788 30263 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:12.393208 30879 raft_consensus.cc:513] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:12.473310 30903 raft_consensus.cc:2747] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250905 20:00:12.473855 30900 log.cc:927] Time spent T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.077s user 0.001s sys 0.000s
W20250905 20:00:12.474792 30896 log.cc:927] Time spent T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.114s user 0.001s sys 0.000s
W20250905 20:00:12.474915 30873 log.cc:927] Time spent T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.589s user 0.006s sys 0.000s
I20250905 20:00:12.482791 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 20:00:12.484196 30901 log.cc:927] Time spent T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
W20250905 20:00:12.485138 30890 log.cc:927] Time spent T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.257s user 0.000s sys 0.001s
W20250905 20:00:12.490917 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 255604us
I20250905 20:00:12.491604 30670 leader_election.cc:290] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:12.464987 30775 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:12.495575 28743 raft_consensus.cc:1238] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:12.496935 30594 consensus_queue.cc:260] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:12.500797 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:12.447033 30721 leader_election.cc:290] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:12.456929 30888 raft_consensus.cc:491] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:12.500042 30832 raft_consensus.cc:491] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
W20250905 20:00:12.520386 30741 consensus_meta.cc:287] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.980s user 0.009s sys 0.015s
I20250905 20:00:12.458206 30891 raft_consensus.cc:3058] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:12.494683 30839 raft_consensus.cc:3053] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Stepping down as leader of term 5
W20250905 20:00:12.522830 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:12.539655 28159 raft_consensus.cc:1273] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250905 20:00:12.539112 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:12.523022 27973 leader_election.cc:304] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
W20250905 20:00:12.525516 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:12.595497 28152 raft_consensus.cc:2466] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 6.
I20250905 20:00:12.596639 28050 leader_election.cc:304] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:12.529886 28079 raft_consensus.cc:3058] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:12.538724 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:12.539359 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:12.598317 28154 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
I20250905 20:00:12.423482 30895 consensus_queue.cc:1046] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.278s
I20250905 20:00:12.425719 30884 raft_consensus.cc:513] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:12.457904 30777 raft_consensus.cc:2802] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:12.590513 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:12.542999 30713 leader_election.cc:290] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:12.521453 30897 consensus_queue.cc:1035] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.088s
I20250905 20:00:12.587524 30832 raft_consensus.cc:3058] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:12.587303 30675 leader_election.cc:290] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:12.651194 28749 raft_consensus.cc:1238] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:12.602901 30839 raft_consensus.cc:738] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:12.649974 30713 consensus_queue.cc:1035] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.009s
I20250905 20:00:12.615821 30777 raft_consensus.cc:491] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:12.621876 30898 raft_consensus.cc:2802] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:12.622720 30893 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:12.669188 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:12.659982 30864 log.cc:927] Time spent T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.053s user 0.000s sys 0.001s
I20250905 20:00:12.579172 30775 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:12.604007 30240 consensus_queue.cc:1046] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.057s
I20250905 20:00:12.587495 30888 raft_consensus.cc:513] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:12.718449 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 110565us
I20250905 20:00:12.652791 30850 raft_consensus.cc:2747] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 20:00:12.666368 30777 raft_consensus.cc:3058] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:12.668396 30672 leader_election.cc:290] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:12.570382 30823 consensus_queue.cc:237] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:12.691350 30883 raft_consensus.cc:491] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:12.694535 30879 leader_election.cc:290] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:12.694900 30824 raft_consensus.cc:513] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:12.601828 30741 raft_consensus.cc:513] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:12.666011 30898 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
W20250905 20:00:12.744113 30892 log.cc:927] Time spent T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.146s user 0.001s sys 0.000s
I20250905 20:00:12.621510 30835 consensus_queue.cc:237] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:12.799383 28002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 2045) took 1315 ms. Trace:
W20250905 20:00:12.745615 30880 log.cc:927] Time spent T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.575s user 0.002s sys 0.000s
W20250905 20:00:12.747525 30913 log.cc:927] Time spent T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.302s user 0.002s sys 0.000s
I20250905 20:00:12.654774 30871 consensus_queue.cc:1046] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.064s
W20250905 20:00:12.747982 30875 consensus_meta.cc:287] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.690s user 0.014s sys 0.004s
I20250905 20:00:12.748447 30612 raft_consensus.cc:3053] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 20:00:12.748378 30858 consensus_queue.cc:1046] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.083s
I20250905 20:00:12.683785 30893 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:12.749627 30882 log.cc:927] Time spent T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.718s user 0.001s sys 0.001s
I20250905 20:00:12.677350 30830 raft_consensus.cc:491] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:12.675740 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 87811.9us
I20250905 20:00:12.775203 28079 raft_consensus.cc:2466] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
I20250905 20:00:12.783612 30594 raft_consensus.cc:3058] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:12.791157 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:12.745141 30756 consensus_meta.cc:287] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.707s user 0.012s sys 0.000s
I20250905 20:00:12.677065 30670 consensus_queue.cc:1046] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.109s
W20250905 20:00:12.814673 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:12.816407 28051 leader_election.cc:304] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:12.818266 28744 raft_consensus.cc:1238] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6. Ops: []
W20250905 20:00:12.818006 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 295509us
I20250905 20:00:12.781514 30883 raft_consensus.cc:513] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:12.837888 30612 raft_consensus.cc:738] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
W20250905 20:00:12.822201 30907 log.cc:927] Time spent T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.140s user 0.000s sys 0.000s
I20250905 20:00:12.822880 30867 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:12.927227 28158 raft_consensus.cc:1238] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 7. Current term is 8. Ops: []
I20250905 20:00:12.927603 30891 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:12.928292 30867 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:12.928658 28749 raft_consensus.cc:1238] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:12.929162 30884 leader_election.cc:290] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:12.939558 28084 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 313) took 1245 ms. Trace:
I20250905 20:00:12.930567 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:12.940114 28084 rpcz_store.cc:276] 0905 20:00:11.694259 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:11.694538 (+ 279us) service_pool.cc:225] Handling call
0905 20:00:11.696827 (+ 2289us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:11.697389 (+ 562us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:11.697452 (+ 63us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:11.698177 (+ 725us) log.cc:844] Serialized 26 byte log entry
0905 20:00:11.882610 (+184433us) raft_consensus.cc:1643] Marking committed up to 7
0905 20:00:11.882667 (+ 57us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:11.882741 (+ 74us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:12.748883 (+866142us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49d85891e 00007fe49d857c66 00007fe49d857888 00007fe49d8612c1 00007fe49d861279 00007fe49d861209
0905 20:00:12.939151 (+190268us) raft_consensus.cc:1683] finished
0905 20:00:12.939183 (+ 32us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:12.939463 (+ 280us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7563,"raft.queue_time_us":9775,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":9763,"spinlock_wait_cycles":23530240,"thread_start_us":142609,"threads_started":1,"wal-append.queue_time_us":287018}
I20250905 20:00:12.943284 30612 consensus_queue.cc:260] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:12.945278 30612 raft_consensus.cc:3058] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:12.938702 30832 raft_consensus.cc:513] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:12.947417 30832 leader_election.cc:290] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:12.876407 30756 raft_consensus.cc:2747] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193) with higher term. Message: Invalid argument: T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6.
I20250905 20:00:12.952368 28010 raft_consensus.cc:1238] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 5. Current term is 6. Ops: []
W20250905 20:00:12.952450 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 136133us
I20250905 20:00:12.953600 28051 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:12.824494 30775 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:12.824636 30912 log.cc:927] Time spent T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.158s user 0.000s sys 0.001s
I20250905 20:00:12.796248 30898 raft_consensus.cc:3058] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:12.826927 30894 raft_consensus.cc:2747] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:12.799945 28002 rpcz_store.cc:276] 0905 20:00:11.475825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:11.497200 (+ 21375us) service_pool.cc:225] Handling call
0905 20:00:11.497561 (+ 361us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:11.527414 (+ 29853us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:11.527454 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:11.565809 (+ 38355us) log.cc:844] Serialized 26 byte log entry
0905 20:00:11.680901 (+115092us) raft_consensus.cc:1643] Marking committed up to 4
0905 20:00:11.716122 (+ 35221us) log.cc:844] Serialized 14 byte log entry
0905 20:00:11.718418 (+ 2296us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:11.718545 (+ 127us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:11.885072 (+166527us) spinlock_profiling.cc:243] Waited 40.3 ms on lock 0x7b180029b240. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49aca43e2 00007fe49aca67b1 00007fe49aca6769 00007fe49aca66f9 00007fe49aca66c1 00007fe49aca59bd 0000000000379254 0000000000379099 00007fe49ac8776a
0905 20:00:12.035319 (+150247us) spinlock_profiling.cc:243] Waited 4.4 ms on lock 0x7b180029b240. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49ac723d6 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49aca450f 00007fe49aca67b1
0905 20:00:12.035734 (+ 415us) spinlock_profiling.cc:243] Waited 38.2 ms on lock 0x7b180029b240. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49aca450f 00007fe49aca67b1 00007fe49aca6769 00007fe49aca66f9 00007fe49aca66c1 00007fe49aca59bd 0000000000379254 0000000000379099 00007fe49ac8776a
0905 20:00:12.474168 (+438434us) spinlock_profiling.cc:243] Waited 14 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49d85891e 00007fe49d857c66 00007fe49d857888 00007fe49d8612c1 00007fe49d861279 00007fe49d861209
0905 20:00:12.790580 (+316412us) raft_consensus.cc:1683] finished
0905 20:00:12.790610 (+ 30us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:12.790791 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1951,"prepare.queue_time_us":36843,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":53,"raft.queue_time_us":2350,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":5908,"spinlock_wait_cycles":222837504,"thread_start_us":63350,"threads_started":1,"wal-append.queue_time_us":146161}
I20250905 20:00:12.973663 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:12.976071 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:12.816121 30879 raft_consensus.cc:491] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:12.828291 30263 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:12.853240 30875 raft_consensus.cc:2747] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:12.858547 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:12.859166 30824 leader_election.cc:290] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:12.860539 28122 leader_election.cc:304] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:12.860925 30893 leader_election.cc:290] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:12.861428 30830 raft_consensus.cc:513] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:12.863438 30741 leader_election.cc:290] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:12.828095 30773 leader_election.cc:290] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421)
W20250905 20:00:12.867642 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 264874us
I20250905 20:00:12.917307 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 20:00:12.917722 30928 log.cc:927] Time spent T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20250905 20:00:13.062268 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 20:00:13.063488 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 71476.2us
I20250905 20:00:12.918928 30883 leader_election.cc:290] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:12.921613 30859 consensus_meta.cc:287] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.974s user 0.007s sys 0.006s
I20250905 20:00:12.856333 30846 raft_consensus.cc:3053] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Stepping down as leader of term 5
I20250905 20:00:12.962103 28051 leader_election.cc:304] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 20:00:13.069955 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 209488us
I20250905 20:00:12.780054 30839 consensus_queue.cc:260] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:12.848470 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:12.985400 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:13.075944 28008 raft_consensus.cc:2466] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
W20250905 20:00:12.992277 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 55994.6us
W20250905 20:00:12.983158 27975 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.601s user 0.011s sys 0.003s
W20250905 20:00:13.077085 27975 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.695s user 0.011s sys 0.004s
I20250905 20:00:12.961194 30672 raft_consensus.cc:2747] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:12.961179 30596 raft_consensus.cc:491] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:12.975628 30660 raft_consensus.cc:2747] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250905 20:00:13.090122 30902 raft_consensus.cc:3053] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 5 LEADER]: Stepping down as leader of term 5
I20250905 20:00:13.090618 30902 raft_consensus.cc:738] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:13.091539 30902 consensus_queue.cc:260] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:12.983469 30866 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.012s user 0.004s sys 0.013s
I20250905 20:00:13.060094 28050 leader_election.cc:304] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
W20250905 20:00:13.093566 30866 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.122s user 0.004s sys 0.013s
I20250905 20:00:13.094991 30883 raft_consensus.cc:2747] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 20:00:13.095305 28050 leader_election.cc:304] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:13.097868 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:13.070678 30875 raft_consensus.cc:2747] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:13.098994 30672 consensus_queue.cc:1035] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 20:00:13.099586 30933 raft_consensus.cc:2802] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:13.100024 30830 leader_election.cc:290] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:13.100816 30902 raft_consensus.cc:3058] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:13.101229 30941 raft_consensus.cc:491] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:13.101789 30941 raft_consensus.cc:513] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:12.977545 30777 raft_consensus.cc:513] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:13.071049 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 109064us
I20250905 20:00:13.071158 28122 leader_election.cc:304] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:13.069406 30773 raft_consensus.cc:491] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:13.050743 28743 raft_consensus.cc:1811] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Handling vote request from an unknown peer cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:12.983284 30879 raft_consensus.cc:513] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:13.120239 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:13.121830 30879 leader_election.cc:290] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:13.122892 28083 raft_consensus.cc:2466] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
I20250905 20:00:13.123667 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:13.088214 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:13.124377 28159 raft_consensus.cc:2466] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
I20250905 20:00:13.125332 28743 raft_consensus.cc:2391] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 6: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:13.125814 30877 raft_consensus.cc:3053] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 LEADER]: Stepping down as leader of term 7
I20250905 20:00:13.126222 30877 raft_consensus.cc:738] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:13.127066 30877 consensus_queue.cc:260] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:13.127208 28051 leader_election.cc:304] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:13.128333 30877 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:13.089546 30596 raft_consensus.cc:513] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:13.089766 30919 log.cc:927] Time spent T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.121s user 0.000s sys 0.000s
I20250905 20:00:13.131641 30903 raft_consensus.cc:2747] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:13.104947 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:13.107735 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:12.991400 30588 consensus_queue.cc:1046] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.084s
I20250905 20:00:13.080744 30670 raft_consensus.cc:2747] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:13.062202 30894 raft_consensus.cc:491] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:13.110930 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:13.069973 30846 raft_consensus.cc:738] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
W20250905 20:00:13.113621 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 752191us
I20250905 20:00:13.130259 28009 raft_consensus.cc:2466] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6.
I20250905 20:00:13.116626 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:13.123888 30861 consensus_meta.cc:287] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.969s user 0.014s sys 0.002s
I20250905 20:00:13.132000 28051 leader_election.cc:304] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:13.145087 28078 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 8.
I20250905 20:00:13.145365 28082 raft_consensus.cc:1238] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 LEADER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:13.153700 28750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40904 (request call id 690) took 1591 ms. Trace:
I20250905 20:00:13.145771 28153 raft_consensus.cc:2466] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 20:00:13.112182 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:13.145365 28154 raft_consensus.cc:2466] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
I20250905 20:00:13.088073 30859 raft_consensus.cc:491] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:13.156082 28076 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:13.149298 28744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40904 (request call id 700) took 1145 ms. Trace:
I20250905 20:00:13.148433 27975 leader_election.cc:304] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:13.108202 30839 raft_consensus.cc:3058] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:13.160233 28717 leader_election.cc:304] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:13.133596 28085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1598) took 1077 ms. Trace:
I20250905 20:00:13.161612 28050 leader_election.cc:304] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:13.156730 28744 rpcz_store.cc:276] 0905 20:00:11.991347 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:11.994531 (+ 3184us) service_pool.cc:225] Handling call
0905 20:00:12.026880 (+ 32349us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:12.873566 (+846686us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:13.136456 (+262890us) spinlock_profiling.cc:243] Waited 791 ms on lock 0x7b5400570dd4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:13.136737 (+ 281us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1819407744}
I20250905 20:00:13.111843 30835 raft_consensus.cc:491] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:13.154156 28750 rpcz_store.cc:276] 0905 20:00:11.562584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:11.563202 (+ 618us) service_pool.cc:225] Handling call
0905 20:00:11.579584 (+ 16382us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:12.370684 (+791100us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:13.153461 (+782777us) spinlock_profiling.cc:243] Waited 782 ms on lock 0x7b540049d4d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:13.153642 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1797682944}
W20250905 20:00:13.135902 30869 consensus_meta.cc:287] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.767s user 0.008s sys 0.004s
I20250905 20:00:13.100051 30933 raft_consensus.cc:491] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250905 20:00:13.166204 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:13.163615 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:13.134882 30885 raft_consensus.cc:3053] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Stepping down as leader of term 4
I20250905 20:00:13.149663 30894 raft_consensus.cc:513] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:13.166142 28007 raft_consensus.cc:1238] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:13.161273 28085 rpcz_store.cc:276] 0905 20:00:12.058494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:12.069624 (+ 11130us) service_pool.cc:225] Handling call
0905 20:00:12.096447 (+ 26823us) raft_consensus.cc:1469] Updating replica for 2 ops
0905 20:00:12.255496 (+159049us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:12.255530 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0905 20:00:12.374277 (+118747us) log.cc:844] Serialized 52 byte log entry
0905 20:00:12.598456 (+224179us) raft_consensus.cc:1643] Marking committed up to 2
0905 20:00:12.964292 (+365836us) log.cc:844] Serialized 14 byte log entry
0905 20:00:12.988929 (+ 24637us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:12.989058 (+ 129us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:13.133382 (+144324us) raft_consensus.cc:1683] finished
0905 20:00:13.133400 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:13.133538 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":114870,"prepare.queue_time_us":146564,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":69100,"raft.run_cpu_time_us":473,"raft.run_wall_time_us":71261,"thread_start_us":525443,"threads_started":3,"wal-append.queue_time_us":474500}
I20250905 20:00:13.133482 30906 raft_consensus.cc:3053] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 LEADER]: Stepping down as leader of term 6
I20250905 20:00:13.174328 28004 raft_consensus.cc:1238] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:13.132669 30596 leader_election.cc:290] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:13.178324 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:13.215958 28159 raft_consensus.cc:2466] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 20:00:13.178867 28005 raft_consensus.cc:3053] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Stepping down as leader of term 5
I20250905 20:00:13.217272 28005 raft_consensus.cc:738] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:13.173964 30933 raft_consensus.cc:3058] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:13.152499 30849 raft_consensus.cc:3053] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 LEADER]: Stepping down as leader of term 6
W20250905 20:00:13.222468 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 90892.1us
I20250905 20:00:13.223472 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:13.225489 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 73600.1us
I20250905 20:00:13.227278 28051 leader_election.cc:304] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:13.228159 28050 leader_election.cc:304] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:13.230369 28717 leader_election.cc:304] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
W20250905 20:00:13.231127 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 51007.5us
W20250905 20:00:13.232908 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 161822us
I20250905 20:00:13.233990 28122 leader_election.cc:304] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:13.164953 30684 raft_consensus.cc:3053] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 20:00:13.159307 30888 leader_election.cc:290] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:13.174369 30885 raft_consensus.cc:738] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:13.173069 30741 raft_consensus.cc:2802] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250905 20:00:13.241336 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 93027us
W20250905 20:00:13.222211 30942 log.cc:927] Time spent T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.122s user 0.000s sys 0.001s
I20250905 20:00:13.228076 28005 consensus_queue.cc:260] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:13.241581 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:13.142987 30773 raft_consensus.cc:513] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:13.226828 30849 raft_consensus.cc:738] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:13.222992 30906 raft_consensus.cc:738] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:13.254000 28122 leader_election.cc:304] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:13.235096 30670 raft_consensus.cc:2802] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:13.248724 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:13.262457 28004 raft_consensus.cc:2466] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 20:00:13.165472 30861 raft_consensus.cc:513] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:13.236752 30935 raft_consensus.cc:2802] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 20:00:13.270949 27975 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:13.271162 30941 leader_election.cc:290] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:13.245342 30860 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.207s user 0.004s sys 0.015s
I20250905 20:00:13.272701 30937 raft_consensus.cc:491] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:13.273316 28051 leader_election.cc:304] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:13.274852 30894 leader_election.cc:290] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:13.254755 30932 raft_consensus.cc:2747] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 20:00:13.235895 30926 raft_consensus.cc:2762] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20250905 20:00:13.238121 30871 consensus_queue.cc:1046] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.055s
I20250905 20:00:13.247237 30721 raft_consensus.cc:2802] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250905 20:00:13.321679 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 72007.5us
I20250905 20:00:13.259166 30938 raft_consensus.cc:2802] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:13.173357 30835 raft_consensus.cc:513] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:13.230706 30588 consensus_queue.cc:1046] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20250905 20:00:13.167811 30859 raft_consensus.cc:513] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:13.236529 30940 raft_consensus.cc:2747] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250905 20:00:13.238879 30684 raft_consensus.cc:738] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:13.270659 30775 consensus_queue.cc:1035] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 20:00:13.254859 30741 raft_consensus.cc:491] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:13.305488 28005 raft_consensus.cc:3058] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:13.239707 30846 consensus_queue.cc:260] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:13.177021 30869 raft_consensus.cc:513] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:13.313298 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:13.313453 28003 raft_consensus.cc:1238] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 3. Current term is 5. Ops: []
I20250905 20:00:13.337527 30721 raft_consensus.cc:491] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:13.314600 27975 leader_election.cc:304] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:13.249845 30263 consensus_queue.cc:1035] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
W20250905 20:00:13.305598 30860 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.268s user 0.005s sys 0.016s
I20250905 20:00:13.370230 30537 consensus_queue.cc:1046] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20250905 20:00:13.370965 30936 log.cc:927] Time spent T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.270s user 0.001s sys 0.000s
I20250905 20:00:13.344939 30849 consensus_queue.cc:260] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:13.376186 30849 raft_consensus.cc:3058] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Advancing to term 7
W20250905 20:00:13.377775 30889 log.cc:927] Time spent T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 1.147s user 0.000s sys 0.004s
I20250905 20:00:13.381946 30891 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 20:00:13.382565 30594 consensus_meta.cc:287] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.510s user 0.011s sys 0.000s
I20250905 20:00:13.382838 28150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1638) took 2033 ms. Trace:
I20250905 20:00:13.382483 30906 consensus_queue.cc:260] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:13.383224 28150 rpcz_store.cc:276] 0905 20:00:11.349121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:11.349292 (+ 171us) service_pool.cc:225] Handling call
0905 20:00:11.349702 (+ 410us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:11.966020 (+616318us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:11.966057 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:11.966623 (+ 566us) log.cc:844] Serialized 26 byte log entry
0905 20:00:12.113697 (+147074us) raft_consensus.cc:1643] Marking committed up to 6
0905 20:00:12.113760 (+ 63us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:12.113889 (+ 129us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:13.377058 (+1263169us) spinlock_profiling.cc:243] Waited 18.8 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49d85891e 00007fe49d857c66 00007fe49d857888 00007fe49d8612c1 00007fe49d861279 00007fe49d861209
0905 20:00:13.382518 (+ 5460us) raft_consensus.cc:1683] finished
0905 20:00:13.382545 (+ 27us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:13.382775 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7702,"raft.queue_time_us":134387,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":19458,"spinlock_wait_cycles":43233536,"thread_start_us":739172,"threads_started":2,"wal-append.queue_time_us":218417}
I20250905 20:00:13.386711 28010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 2046) took 1850 ms. Trace:
I20250905 20:00:13.387203 28010 rpcz_store.cc:276] 0905 20:00:11.536888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:11.547615 (+ 10727us) service_pool.cc:225] Handling call
0905 20:00:11.996312 (+448697us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:12.952979 (+956667us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:13.124207 (+171228us) spinlock_profiling.cc:243] Waited 956 ms on lock 0x7b540038f754. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:13.386408 (+262201us) spinlock_profiling.cc:243] Waited 423 ms on lock 0x7b540038f750. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:13.386629 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3170804352}
I20250905 20:00:13.388265 30777 leader_election.cc:290] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:13.389657 30939 consensus_queue.cc:1035] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.141s
I20250905 20:00:13.391655 30939 raft_consensus.cc:2747] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250905 20:00:13.345384 30830 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:13.392963 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 68862.1us
I20250905 20:00:13.324180 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:13.393997 28159 raft_consensus.cc:3058] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 6
I20250905 20:00:13.344636 30938 raft_consensus.cc:491] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:13.340983 28749 raft_consensus.cc:1238] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [term 7 LEADER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:13.305440 30935 raft_consensus.cc:491] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:13.292440 30930 raft_consensus.cc:2802] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election won for term 6
I20250905 20:00:13.437233 28157 raft_consensus.cc:2391] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 7: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
W20250905 20:00:13.437454 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 123015us
I20250905 20:00:13.360219 30721 raft_consensus.cc:3058] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:13.315716 30148 raft_consensus.cc:2747] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250905 20:00:13.314777 28051 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 20:00:13.360991 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 49998.1us
I20250905 20:00:13.322638 30931 raft_consensus.cc:2747] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250905 20:00:13.310988 30937 raft_consensus.cc:513] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:13.365983 30684 consensus_queue.cc:260] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:13.366837 30773 leader_election.cc:290] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:13.431890 30938 raft_consensus.cc:3058] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
W20250905 20:00:13.367745 30922 log.cc:927] Time spent T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20250905 20:00:13.447264 30930 raft_consensus.cc:695] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:13.368355 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:13.455518 30930 consensus_queue.cc:237] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:13.456386 28006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 346) took 1799 ms. Trace:
I20250905 20:00:13.456779 28006 rpcz_store.cc:276] 0905 20:00:11.657380 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:12.198874 (+541494us) service_pool.cc:225] Handling call
0905 20:00:12.199689 (+ 815us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:12.215753 (+ 16064us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:12.216013 (+ 260us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:12.477033 (+261020us) spinlock_profiling.cc:243] Waited 44.4 ms on lock 0x7b2c000014c8. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49c5bc9b1 00007fe49c5cbfe2 00007fe49dc55021 00007fe49dc193fa 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32
0905 20:00:12.477507 (+ 474us) log.cc:844] Serialized 26 byte log entry
0905 20:00:12.562903 (+ 85396us) raft_consensus.cc:1643] Marking committed up to 7
0905 20:00:12.562963 (+ 60us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:12.563064 (+ 101us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:13.456062 (+892998us) raft_consensus.cc:1683] finished
0905 20:00:13.456083 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:13.456323 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11577,"raft.queue_time_us":16453,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":86,"spinlock_wait_cycles":102052096,"thread_start_us":84918,"threads_started":1,"wal-append.queue_time_us":762448}
I20250905 20:00:13.369346 30240 raft_consensus.cc:3053] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 LEADER]: Stepping down as leader of term 5
I20250905 20:00:13.458577 30240 raft_consensus.cc:738] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
W20250905 20:00:13.458745 30948 log.cc:927] Time spent T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.111s user 0.003s sys 0.000s
I20250905 20:00:13.459868 28003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 2062) took 1488 ms. Trace:
I20250905 20:00:13.292635 30670 raft_consensus.cc:491] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:13.394891 27973 leader_election.cc:304] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:13.460566 30670 raft_consensus.cc:3058] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:13.390195 30835 leader_election.cc:290] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:13.349088 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:13.366098 30846 raft_consensus.cc:3058] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
W20250905 20:00:13.345965 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:13.476588 28159 raft_consensus.cc:2466] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 6.
I20250905 20:00:13.478762 28743 raft_consensus.cc:2366] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5.
W20250905 20:00:13.479341 28008 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 2065) took 1500 ms (client timeout 1938 ms). Trace:
W20250905 20:00:13.479776 28008 rpcz_store.cc:269] 0905 20:00:11.971958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:12.113170 (+141212us) service_pool.cc:225] Handling call
0905 20:00:13.461294 (+1348124us) spinlock_profiling.cc:243] Waited 961 ms on lock 0x7b54000104d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:13.472583 (+ 11289us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2211277696}
I20250905 20:00:13.439419 27975 leader_election.cc:304] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
W20250905 20:00:13.441072 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:13.483500 30956 raft_consensus.cc:2802] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election won for term 6
W20250905 20:00:13.111464 30840 consensus_queue.cc:1162] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7e9c9f591f05464f9b097bf003d14f4a" responder_term: 6 status { last_received { term: 5 index: 7 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250905 20:00:13.484181 30956 raft_consensus.cc:695] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:13.484697 30940 raft_consensus.cc:2747] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250905 20:00:13.484902 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 45585.2us
I20250905 20:00:13.452361 30684 raft_consensus.cc:3058] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:13.343015 30885 consensus_queue.cc:260] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:13.486836 30885 raft_consensus.cc:3058] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:13.487105 30869 leader_election.cc:290] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:13.451476 30925 consensus_queue.cc:1035] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.013s
I20250905 20:00:13.355067 30741 raft_consensus.cc:3058] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:13.460374 28003 rpcz_store.cc:276] 0905 20:00:11.971016 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:12.210202 (+239186us) service_pool.cc:225] Handling call
0905 20:00:12.268383 (+ 58181us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:13.356361 (+1087978us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:13.459108 (+102747us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400291b54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:13.459660 (+ 552us) spinlock_profiling.cc:243] Waited 36 ms on lock 0x7b5400291b50. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:13.459806 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2485653504}
W20250905 20:00:13.489033 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 174353us
I20250905 20:00:13.489198 27975 leader_election.cc:304] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:13.489931 28051 leader_election.cc:304] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:13.490939 30773 raft_consensus.cc:2747] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:13.491463 28051 leader_election.cc:400] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Vote denied by peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421) with higher term. Message: Invalid argument: T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 5. Current term is 6.
I20250905 20:00:13.491775 28051 leader_election.cc:403] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20250905 20:00:13.492122 30940 raft_consensus.cc:2802] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 20:00:13.492762 30940 raft_consensus.cc:695] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:13.494324 30940 consensus_queue.cc:237] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:13.494985 30773 raft_consensus.cc:2747] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421) with higher term. Message: Invalid argument: T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 5. Current term is 6.
I20250905 20:00:13.492941 28051 leader_election.cc:304] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:13.464696 30859 leader_election.cc:290] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:13.465693 30898 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:13.466599 30937 leader_election.cc:290] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:13.437492 30935 raft_consensus.cc:3058] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:13.470975 28001 raft_consensus.cc:1273] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20250905 20:00:13.472011 30612 consensus_meta.cc:287] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.526s user 0.009s sys 0.004s
I20250905 20:00:13.472024 28158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39586 (request call id 343) took 1041 ms. Trace:
W20250905 20:00:13.472821 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 78033.4us
I20250905 20:00:13.485141 30956 consensus_queue.cc:237] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:13.463387 30946 raft_consensus.cc:2747] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:13.459496 30240 consensus_queue.cc:260] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:13.476482 30902 raft_consensus.cc:2747] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20250905 20:00:13.505072 30951 log.cc:927] Time spent T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.158s user 0.001s sys 0.000s
W20250905 20:00:13.506621 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:13.507889 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:13.554145 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:13.544168 30824 raft_consensus.cc:491] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:13.556565 30824 raft_consensus.cc:513] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:13.431769 30830 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:13.509331 28050 leader_election.cc:304] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:13.520021 28156 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1642) took 1350 ms. Trace:
I20250905 20:00:13.519029 28005 raft_consensus.cc:1273] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250905 20:00:13.525866 28123 leader_election.cc:400] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299) with higher term. Message: Invalid argument: T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5.
I20250905 20:00:13.567098 28123 leader_election.cc:403] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20250905 20:00:13.529286 28080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1607) took 1303 ms. Trace:
I20250905 20:00:13.567382 28157 raft_consensus.cc:1273] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20250905 20:00:13.567790 28080 rpcz_store.cc:276] 0905 20:00:12.225557 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:12.451970 (+226413us) service_pool.cc:225] Handling call
0905 20:00:12.468152 (+ 16182us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:12.504106 (+ 35954us) raft_consensus.cc:1532] Early marking committed up to index 3
0905 20:00:12.566726 (+ 62620us) log.cc:844] Serialized 14 byte log entry
0905 20:00:12.758074 (+191348us) log.cc:844] Serialized 14 byte log entry
0905 20:00:12.758490 (+ 416us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:12.758545 (+ 55us) raft_consensus.cc:1643] Marking committed up to 3
0905 20:00:12.758597 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:13.375730 (+617133us) spinlock_profiling.cc:243] Waited 2.32 ms on lock 0x7b1800327100. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49aca450f 00007fe49aca67b1 00007fe49aca6769 00007fe49aca66f9 00007fe49aca66c1 00007fe49aca59bd 0000000000379254 0000000000379099 00007fe49ac8776a
0905 20:00:13.392057 (+ 16327us) spinlock_profiling.cc:243] Waited 5.76 ms on lock 0x7b1800327100. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49aca450f 00007fe49aca67b1 00007fe49aca6769 00007fe49aca66f9 00007fe49aca66c1 00007fe49aca59bd 0000000000379254 0000000000379099 00007fe49ac8776a
0905 20:00:13.529044 (+136987us) spinlock_profiling.cc:243] Waited 35.6 ms on lock 0x7b54003a9154. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:13.529103 (+ 59us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:13.529227 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":135359,"prepare.queue_time_us":288825,"prepare.run_cpu_time_us":109,"prepare.run_wall_time_us":3182,"raft.queue_time_us":67062,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":12822,"spinlock_wait_cycles":100513280,"thread_start_us":89273,"threads_started":2,"wal-append.queue_time_us":235983}
W20250905 20:00:13.526216 30887 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 1.295s user 0.000s sys 0.029s
I20250905 20:00:13.535106 28158 rpcz_store.cc:276] 0905 20:00:12.430621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:12.430788 (+ 167us) service_pool.cc:225] Handling call
0905 20:00:12.446085 (+ 15297us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:12.927858 (+481773us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:13.471760 (+543902us) spinlock_profiling.cc:243] Waited 481 ms on lock 0x7b5400200b54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:13.471960 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1105820544}
W20250905 20:00:13.542994 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:13.510136 30877 raft_consensus.cc:491] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:13.548736 30741 raft_consensus.cc:513] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:13.563448 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 70568.6us
I20250905 20:00:13.535166 30937 raft_consensus.cc:2747] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:13.530237 28083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1613) took 1008 ms. Trace:
I20250905 20:00:13.565552 28156 rpcz_store.cc:276] 0905 20:00:12.169976 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:12.171374 (+ 1398us) service_pool.cc:225] Handling call
0905 20:00:12.204105 (+ 32731us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:12.415824 (+211719us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:13.519765 (+1103941us) spinlock_profiling.cc:243] Waited 39.1 ms on lock 0x7b5400177854. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:13.519950 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":90049024}
W20250905 20:00:13.570942 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 45286.9us
W20250905 20:00:13.564621 30844 consensus_meta.cc:287] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 1.145s user 0.014s sys 0.000s
I20250905 20:00:13.584935 27973 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 20:00:13.575706 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 66539.4us
I20250905 20:00:13.584699 28083 rpcz_store.cc:276] 0905 20:00:12.521717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:12.741862 (+220145us) service_pool.cc:225] Handling call
0905 20:00:13.530079 (+788217us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b540023bc54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:13.530179 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":584724736}
I20250905 20:00:13.573714 27975 leader_election.cc:304] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:13.579870 28005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 2085) took 1531 ms. Trace:
I20250905 20:00:13.549765 30939 consensus_queue.cc:1046] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.029s
I20250905 20:00:13.583022 28051 leader_election.cc:304] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:13.583029 28745 raft_consensus.cc:1273] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250905 20:00:13.603679 27975 leader_election.cc:304] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:13.605916 27975 leader_election.cc:304] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:13.600353 28005 rpcz_store.cc:276] 0905 20:00:12.048949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:12.217244 (+168295us) service_pool.cc:225] Handling call
0905 20:00:12.217577 (+ 333us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:13.178744 (+961167us) spinlock_profiling.cc:243] Waited 8.51 ms on lock 0x7b540036aea0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dba8550 00007fe49dc1b170 00007fe49dc1c1e1 00007fe49dc18d4f 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c
0905 20:00:13.579479 (+400735us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:13.579791 (+ 312us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19573504}
W20250905 20:00:13.585016 30690 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20250905 20:00:13.576728 30612 raft_consensus.cc:2747] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299) with higher term. Message: Invalid argument: T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5.
W20250905 20:00:13.584138 30887 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 1.353s user 0.000s sys 0.030s
I20250905 20:00:13.617830 28744 raft_consensus.cc:1238] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20250905 20:00:13.583823 30883 raft_consensus.cc:2802] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election won for term 6
I20250905 20:00:13.623157 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:13.607826 30940 raft_consensus.cc:2802] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250905 20:00:13.623878 28156 raft_consensus.cc:2466] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 20:00:13.608017 30775 raft_consensus.cc:2747] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250905 20:00:13.612226 30588 raft_consensus.cc:2747] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250905 20:00:13.624241 28051 leader_election.cc:304] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:13.615139 30824 leader_election.cc:290] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:13.584455 30877 raft_consensus.cc:513] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:13.601233 30597 consensus_queue.cc:1046] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.023s
I20250905 20:00:13.634886 28717 leader_election.cc:304] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:13.622277 30937 raft_consensus.cc:2802] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 20:00:13.641242 27973 leader_election.cc:304] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:13.643949 30741 leader_election.cc:290] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:13.644690 30898 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:13.645179 30857 consensus_queue.cc:1035] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20250905 20:00:13.645558 30721 raft_consensus.cc:513] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:13.647229 30954 consensus_queue.cc:1035] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
W20250905 20:00:13.648092 30945 log.cc:927] Time spent T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.232s user 0.000s sys 0.001s
I20250905 20:00:13.648274 30775 consensus_queue.cc:1035] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 20:00:13.690316 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:13.648552 30833 consensus_queue.cc:1035] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.051s
I20250905 20:00:13.602315 30933 raft_consensus.cc:513] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:13.660154 28051 leader_election.cc:304] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:13.628783 30883 raft_consensus.cc:695] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:13.667814 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:13.703742 28157 raft_consensus.cc:3058] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:13.670749 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:13.706790 28083 raft_consensus.cc:2466] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 8.
I20250905 20:00:13.671149 28718 leader_election.cc:304] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:13.671911 30897 consensus_queue.cc:1046] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250905 20:00:13.673480 28125 leader_election.cc:304] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:13.677065 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:13.709847 28082 raft_consensus.cc:2408] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250905 20:00:13.627053 30940 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:13.711004 30940 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Advancing to term 9
I20250905 20:00:13.641912 30939 consensus_queue.cc:1035] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 20:00:13.712860 28001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 2088) took 1549 ms. Trace:
I20250905 20:00:13.713425 28001 rpcz_store.cc:276] 0905 20:00:12.163689 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:12.253539 (+ 89850us) service_pool.cc:225] Handling call
0905 20:00:13.462460 (+1208921us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:13.526951 (+ 64491us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:13.712581 (+185630us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54000104d0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:13.712780 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2532748672}
I20250905 20:00:13.715226 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:13.716790 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:13.691511 28150 raft_consensus.cc:1273] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250905 20:00:13.688194 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:13.720448 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:13.720733 28150 raft_consensus.cc:1238] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20250905 20:00:13.692811 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:13.722031 28006 raft_consensus.cc:2466] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 20:00:13.718976 28001 raft_consensus.cc:1273] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250905 20:00:13.723606 28006 raft_consensus.cc:1238] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20250905 20:00:13.693225 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:13.686316 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:13.693925 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:13.726364 28152 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 8.
I20250905 20:00:13.696722 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:13.727921 28005 raft_consensus.cc:2466] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 20:00:13.638710 30849 raft_consensus.cc:2747] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250905 20:00:13.700388 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:13.729208 28009 raft_consensus.cc:3058] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:13.690625 30824 raft_consensus.cc:2802] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 20:00:13.729736 30824 raft_consensus.cc:491] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:13.730144 30824 raft_consensus.cc:3058] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Advancing to term 5
W20250905 20:00:13.730300 30952 log.cc:927] Time spent T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.133s user 0.000s sys 0.001s
I20250905 20:00:13.699618 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:13.701876 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:13.629967 30612 consensus_queue.cc:1046] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.066s
I20250905 20:00:13.735358 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:13.738267 28151 raft_consensus.cc:1273] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250905 20:00:13.749444 28008 raft_consensus.cc:1238] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:13.679412 30902 raft_consensus.cc:2747] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250905 20:00:13.763151 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:13.771030 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 99948us
I20250905 20:00:13.774264 30823 raft_consensus.cc:2802] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader election won for term 6
I20250905 20:00:13.683779 30963 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:13.779346 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:13.784148 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 110773us
I20250905 20:00:13.735975 30675 raft_consensus.cc:3058] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
W20250905 20:00:13.791173 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 131123us
I20250905 20:00:13.798156 28051 leader_election.cc:304] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:13.789997 28125 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:13.792521 28151 raft_consensus.cc:1238] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:13.800693 28125 leader_election.cc:304] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 20:00:13.805428 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:13.805303 28051 leader_election.cc:304] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:13.814574 27975 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:13.825500 28009 raft_consensus.cc:2466] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6.
I20250905 20:00:13.830899 30830 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:13.830981 30850 raft_consensus.cc:491] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:13.831651 30850 raft_consensus.cc:513] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:13.791527 30963 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:13.833876 30850 leader_election.cc:290] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:13.783622 30866 raft_consensus.cc:491] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:13.775156 30967 raft_consensus.cc:2802] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:13.844811 28157 raft_consensus.cc:2466] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6.
I20250905 20:00:13.794083 30897 raft_consensus.cc:2747] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250905 20:00:13.845983 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:13.846541 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:13.851178 28717 leader_election.cc:304] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:13.851826 28151 raft_consensus.cc:3058] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:13.796139 30721 leader_election.cc:290] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:13.852582 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:13.807709 30969 raft_consensus.cc:491] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:13.797230 30823 raft_consensus.cc:695] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:13.814502 30941 raft_consensus.cc:2747] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250905 20:00:13.858671 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:13.817826 30835 raft_consensus.cc:2802] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:13.780292 30925 consensus_queue.cc:1035] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
I20250905 20:00:13.826714 30875 consensus_queue.cc:1046] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.045s
I20250905 20:00:13.874718 30953 consensus_queue.cc:1035] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 20:00:13.855346 30866 raft_consensus.cc:513] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:13.876677 30938 raft_consensus.cc:513] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 20:00:13.877836 30866 leader_election.cc:290] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:13.834327 30963 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:13.879779 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 65318.9us
I20250905 20:00:13.883910 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 20:00:13.893810 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:13.850967 30967 raft_consensus.cc:491] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:13.897814 30967 raft_consensus.cc:3058] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:13.898243 30933 leader_election.cc:290] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:13.826524 30891 consensus_queue.cc:1035] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.046s
I20250905 20:00:13.861428 30833 raft_consensus.cc:2802] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader election won for term 6
I20250905 20:00:13.900529 30833 raft_consensus.cc:695] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:13.855787 30902 raft_consensus.cc:3058] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:13.905630 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:13.908066 28157 raft_consensus.cc:3058] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:13.858829 30773 raft_consensus.cc:2747] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:13.901677 30833 consensus_queue.cc:237] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:13.862393 30835 raft_consensus.cc:491] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250905 20:00:13.916682 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 98406.5us
I20250905 20:00:13.923324 27973 leader_election.cc:304] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:13.937318 28157 raft_consensus.cc:2466] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 7.
I20250905 20:00:13.924952 30835 raft_consensus.cc:3058] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
W20250905 20:00:13.947971 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 64805.5us
I20250905 20:00:13.932658 30960 raft_consensus.cc:3058] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:13.949694 28050 leader_election.cc:304] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:13.957856 27973 leader_election.cc:304] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:13.870088 30883 consensus_queue.cc:237] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:13.936690 30877 leader_election.cc:290] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:13.942703 30938 leader_election.cc:290] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421)
I20250905 20:00:13.868429 30969 raft_consensus.cc:513] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:13.964689 30775 raft_consensus.cc:2802] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20250905 20:00:13.969622 30670 consensus_meta.cc:287] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.503s user 0.006s sys 0.006s
I20250905 20:00:13.989235 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:13.991444 28009 raft_consensus.cc:1273] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250905 20:00:13.994063 28082 raft_consensus.cc:1238] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:13.975683 30773 raft_consensus.cc:2802] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader election won for term 7
I20250905 20:00:13.998181 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:14.002499 28006 raft_consensus.cc:3058] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:13.999550 28152 raft_consensus.cc:1238] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:14.007563 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:14.008369 28007 raft_consensus.cc:2391] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 6: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:14.008548 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:14.009332 28076 raft_consensus.cc:3058] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:13.957391 30823 consensus_queue.cc:237] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:14.016286 28005 raft_consensus.cc:1273] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250905 20:00:14.021212 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:14.021871 28009 raft_consensus.cc:2466] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 20:00:13.994158 30839 raft_consensus.cc:2747] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:14.026615 30937 raft_consensus.cc:695] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:14.024236 28125 leader_election.cc:304] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:14.028345 30869 raft_consensus.cc:2802] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:14.029062 30869 raft_consensus.cc:491] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:14.028290 30937 consensus_queue.cc:237] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:14.029480 30869 raft_consensus.cc:3058] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
W20250905 20:00:13.987068 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 181837us
I20250905 20:00:14.039822 30875 raft_consensus.cc:491] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:14.040745 30875 raft_consensus.cc:513] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:14.039832 27973 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:14.043481 27973 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:14.046128 28083 raft_consensus.cc:1238] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:14.049059 30148 consensus_queue.cc:1046] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 20:00:14.050870 30862 raft_consensus.cc:3053] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Stepping down as leader of term 5
I20250905 20:00:14.051405 30862 raft_consensus.cc:738] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:14.051681 30875 leader_election.cc:290] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:13.958854 30968 consensus_queue.cc:1035] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.053s
I20250905 20:00:13.993932 30775 raft_consensus.cc:491] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:14.057170 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:13.993427 30939 raft_consensus.cc:491] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:14.019239 30861 leader_election.cc:290] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:14.060631 27975 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:14.084887 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:14.093546 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:14.082098 28076 raft_consensus.cc:2466] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:14.052520 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:14.055302 28051 leader_election.cc:304] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:14.102808 27973 leader_election.cc:304] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
W20250905 20:00:14.106357 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 51188.4us
I20250905 20:00:14.106554 28006 raft_consensus.cc:2466] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6.
I20250905 20:00:14.116694 28051 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:14.116891 28718 leader_election.cc:304] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:13.996655 30670 raft_consensus.cc:513] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:14.000723 30877 consensus_queue.cc:1046] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.092s
I20250905 20:00:14.025310 30773 raft_consensus.cc:695] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:14.007738 30537 consensus_queue.cc:1035] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.101s
W20250905 20:00:14.109131 30935 consensus_meta.cc:287] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.565s user 0.014s sys 0.000s
I20250905 20:00:14.092650 30775 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Advancing to term 9
I20250905 20:00:14.015362 30977 consensus_queue.cc:1046] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.108s
I20250905 20:00:14.054193 30926 consensus_queue.cc:1035] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.073s
I20250905 20:00:14.108160 30957 consensus_queue.cc:1046] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 2, Time since last communication: 0.041s
I20250905 20:00:14.052778 30862 consensus_queue.cc:260] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
W20250905 20:00:14.117621 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 310374us
I20250905 20:00:14.274250 28151 raft_consensus.cc:2466] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6.
W20250905 20:00:14.145212 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:14.319708 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:14.151291 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:14.322032 28152 raft_consensus.cc:2466] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6.
W20250905 20:00:14.357674 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:14.233829 30684 consensus_meta.cc:287] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.742s user 0.007s sys 0.006s
I20250905 20:00:14.122291 30969 leader_election.cc:290] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:14.135896 30846 consensus_meta.cc:287] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.651s user 0.012s sys 0.000s
W20250905 20:00:14.417244 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 300604us
I20250905 20:00:14.236138 30906 raft_consensus.cc:3058] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Advancing to term 7
W20250905 20:00:14.460704 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 186561us
I20250905 20:00:14.107960 30939 raft_consensus.cc:513] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:14.222446 30839 raft_consensus.cc:2802] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election won for term 5
I20250905 20:00:14.132001 30971 raft_consensus.cc:2802] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader election won for term 6
I20250905 20:00:14.241312 30938 raft_consensus.cc:491] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:14.154498 30660 raft_consensus.cc:2747] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250905 20:00:14.207679 30263 raft_consensus.cc:3053] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 LEADER]: Stepping down as leader of term 6
I20250905 20:00:14.543557 30263 raft_consensus.cc:738] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:14.544807 30263 consensus_queue.cc:260] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:14.237061 30663 raft_consensus.cc:3053] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 LEADER]: Stepping down as leader of term 8
I20250905 20:00:14.546885 30663 raft_consensus.cc:738] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:14.548027 30663 consensus_queue.cc:260] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:14.549356 30663 raft_consensus.cc:3058] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Advancing to term 9
W20250905 20:00:14.558308 30987 log.cc:927] Time spent T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.317s user 0.000s sys 0.001s
I20250905 20:00:14.171782 30877 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:14.446601 28004 raft_consensus.cc:2408] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250905 20:00:14.154158 30975 raft_consensus.cc:2747] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:14.455952 28007 raft_consensus.cc:2466] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6.
I20250905 20:00:14.123920 30869 raft_consensus.cc:513] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:14.159314 30824 raft_consensus.cc:513] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:14.456375 30885 raft_consensus.cc:491] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:14.518838 30993 log.cc:927] Time spent T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20250905 20:00:14.479501 30969 raft_consensus.cc:491] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:14.535725 30597 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:14.597266 28749 raft_consensus.cc:1238] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
W20250905 20:00:14.599094 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 343282us
I20250905 20:00:14.389134 30935 raft_consensus.cc:513] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:14.546277 30263 raft_consensus.cc:3058] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Advancing to term 7
W20250905 20:00:14.578701 30940 consensus_meta.cc:287] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.867s user 0.013s sys 0.002s
I20250905 20:00:14.663262 28717 leader_election.cc:304] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
W20250905 20:00:14.630725 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 389639us
I20250905 20:00:14.463851 30937 raft_consensus.cc:491] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:14.645627 28151 raft_consensus.cc:1273] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250905 20:00:14.517204 28051 leader_election.cc:304] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:14.598862 30860 raft_consensus.cc:491] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:14.648968 30862 raft_consensus.cc:3058] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:14.730554 30860 raft_consensus.cc:513] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:14.732877 30860 leader_election.cc:290] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:14.734480 28010 raft_consensus.cc:1273] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20250905 20:00:14.630704 30773 consensus_queue.cc:237] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:14.727540 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 64525.8us
I20250905 20:00:14.727025 28745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40926 (request call id 509) took 1227 ms. Trace:
I20250905 20:00:14.727701 30937 raft_consensus.cc:513] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:14.728101 30883 raft_consensus.cc:2747] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 20:00:14.728569 30833 raft_consensus.cc:2802] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:14.603757 30938 raft_consensus.cc:513] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:14.729934 30994 log.cc:927] Time spent T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.222s user 0.000s sys 0.000s
I20250905 20:00:14.623450 30971 raft_consensus.cc:695] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:14.741679 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:14.741410 28745 rpcz_store.cc:276] 0905 20:00:13.499269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:13.499469 (+ 200us) service_pool.cc:225] Handling call
0905 20:00:13.503371 (+ 3902us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:13.602169 (+ 98798us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:14.726743 (+1124574us) spinlock_profiling.cc:243] Waited 79.3 ms on lock 0x7b540053d254. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:14.726940 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":182406016}
I20250905 20:00:14.601359 30839 raft_consensus.cc:695] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
W20250905 20:00:14.660858 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:14.748147 28150 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.621s user 0.015s sys 0.067s
W20250905 20:00:14.763309 28150 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.637s user 0.016s sys 0.067s
W20250905 20:00:14.763392 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:14.645866 30537 raft_consensus.cc:491] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:14.764267 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:14.766849 28744 raft_consensus.cc:1238] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:14.725147 30975 raft_consensus.cc:491] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:14.725471 30670 leader_election.cc:290] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:14.769160 28150 raft_consensus.cc:1238] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 7. Current term is 8. Ops: []
I20250905 20:00:14.759450 28749 raft_consensus.cc:2466] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
W20250905 20:00:14.785396 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:14.750121 30833 raft_consensus.cc:491] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:14.641889 30885 raft_consensus.cc:513] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:14.630216 30877 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:14.673326 30969 raft_consensus.cc:513] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:14.794178 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 277043us
I20250905 20:00:14.672695 30597 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:14.819653 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:14.742561 30940 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:14.783326 30946 raft_consensus.cc:491] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:14.794091 30833 raft_consensus.cc:3058] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Advancing to term 7
W20250905 20:00:14.816160 30902 consensus_meta.cc:287] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.893s user 0.014s sys 0.000s
I20250905 20:00:14.860205 28051 leader_election.cc:304] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:14.832366 30670 raft_consensus.cc:491] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:14.786381 30537 raft_consensus.cc:513] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:14.866997 28080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1619) took 1192 ms. Trace:
I20250905 20:00:14.872252 28080 rpcz_store.cc:276] 0905 20:00:13.674533 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:13.674714 (+ 181us) service_pool.cc:225] Handling call
0905 20:00:13.675140 (+ 426us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:13.988101 (+312961us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:13.988150 (+ 49us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:14.023183 (+ 35033us) log.cc:844] Serialized 26 byte log entry
0905 20:00:14.396387 (+373204us) raft_consensus.cc:1643] Marking committed up to 2
0905 20:00:14.396441 (+ 54us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:14.396536 (+ 95us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:14.866692 (+470156us) raft_consensus.cc:1683] finished
0905 20:00:14.866724 (+ 32us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:14.866922 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":66745,"raft.queue_time_us":99448,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":16964,"thread_start_us":612039,"threads_started":2,"wal-append.queue_time_us":822434}
W20250905 20:00:14.869307 30675 consensus_meta.cc:287] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 1.064s user 0.011s sys 0.000s
I20250905 20:00:14.861171 30902 raft_consensus.cc:2747] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 20:00:14.869697 28155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1658) took 1102 ms. Trace:
I20250905 20:00:14.871018 28051 leader_election.cc:304] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:14.789170 30975 raft_consensus.cc:513] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:14.854768 31002 raft_consensus.cc:491] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:14.887881 28155 rpcz_store.cc:276] 0905 20:00:13.767453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:13.767641 (+ 188us) service_pool.cc:225] Handling call
0905 20:00:13.792323 (+ 24682us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:13.793345 (+ 1022us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:13.793386 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:13.794786 (+ 1400us) log.cc:844] Serialized 26 byte log entry
0905 20:00:13.986962 (+192176us) raft_consensus.cc:1643] Marking committed up to 3
0905 20:00:14.127117 (+140155us) log.cc:844] Serialized 14 byte log entry
0905 20:00:14.172634 (+ 45517us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:14.172740 (+ 106us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:14.869470 (+696730us) raft_consensus.cc:1683] finished
0905 20:00:14.869501 (+ 31us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:14.869639 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56674,"prepare.queue_time_us":188195,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":7405,"raft.queue_time_us":45679,"raft.run_cpu_time_us":320,"raft.run_wall_time_us":25450,"thread_start_us":225537,"threads_started":2,"wal-append.queue_time_us":907446}
I20250905 20:00:14.889806 30675 raft_consensus.cc:2747] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:14.888008 30974 consensus_queue.cc:1046] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.006s
I20250905 20:00:14.892697 30902 raft_consensus.cc:2747] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250905 20:00:14.871011 30869 leader_election.cc:290] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:14.853020 30946 raft_consensus.cc:513] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:14.922930 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:14.970688 28080 raft_consensus.cc:2466] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 20:00:14.972106 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:14.887190 30660 raft_consensus.cc:2747] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250905 20:00:14.980295 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:14.906100 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:14.929126 28004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 357) took 1233 ms. Trace:
I20250905 20:00:14.922307 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:14.924139 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:14.923732 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:14.952574 28157 raft_consensus.cc:1273] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20250905 20:00:14.928103 30992 log.cc:927] Time spent T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.127s user 0.000s sys 0.001s
I20250905 20:00:14.922593 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:14.979615 30960 consensus_meta.cc:287] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 1.019s user 0.013s sys 0.000s
I20250905 20:00:14.871215 30971 consensus_queue.cc:237] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:14.993108 30960 raft_consensus.cc:2747] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:14.994347 30869 consensus_queue.cc:1035] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250905 20:00:14.996433 28083 raft_consensus.cc:2391] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 6: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:14.997550 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:14.998776 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:14.999648 28083 raft_consensus.cc:2408] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250905 20:00:14.890347 30977 raft_consensus.cc:491] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:15.002729 30977 raft_consensus.cc:513] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:15.005425 30977 leader_election.cc:290] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:15.006305 28076 raft_consensus.cc:1273] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250905 20:00:14.984311 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:14.889176 30670 raft_consensus.cc:513] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:15.007140 30935 leader_election.cc:290] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:15.008452 30670 leader_election.cc:290] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:15.009827 28076 raft_consensus.cc:1238] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:15.011691 30537 leader_election.cc:290] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:14.900702 30998 consensus_queue.cc:1035] T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.124s
I20250905 20:00:14.900974 30984 raft_consensus.cc:491] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:15.016327 30984 raft_consensus.cc:513] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:15.018141 30885 leader_election.cc:290] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:15.019074 30969 leader_election.cc:290] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:15.022078 30975 leader_election.cc:290] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:15.023248 30597 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:15.024829 30835 consensus_meta.cc:287] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.066s user 0.007s sys 0.004s
I20250905 20:00:15.025261 30835 raft_consensus.cc:513] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:14.984850 28078 raft_consensus.cc:3058] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:14.925978 30940 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 9 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:14.983152 28004 rpcz_store.cc:276] 0905 20:00:13.697100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:13.701203 (+ 4103us) service_pool.cc:225] Handling call
0905 20:00:14.928924 (+1227721us) spinlock_profiling.cc:243] Waited 685 ms on lock 0x7b54003845d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:14.929064 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1576462592}
I20250905 20:00:15.029959 28081 raft_consensus.cc:1238] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: []
W20250905 20:00:14.899873 30823 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.877s user 0.031s sys 0.075s
I20250905 20:00:14.969478 30824 leader_election.cc:290] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:14.984426 28082 raft_consensus.cc:2374] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 9. Re-sending same reply.
I20250905 20:00:14.924909 30937 leader_election.cc:290] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:14.986129 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:15.048768 28080 raft_consensus.cc:2391] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 6: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:14.886971 30883 consensus_queue.cc:1035] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.098s
I20250905 20:00:14.926096 30939 leader_election.cc:290] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:14.982463 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:15.056677 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:15.056983 28150 raft_consensus.cc:2391] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 6: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:15.036581 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:15.057309 28081 raft_consensus.cc:3058] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:14.929544 30938 leader_election.cc:290] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:15.057328 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 20:00:15.037652 30967 consensus_meta.cc:287] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 1.122s user 0.013s sys 0.000s
I20250905 20:00:15.055568 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:15.047569 28077 raft_consensus.cc:1238] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20250905 20:00:15.058343 28749 raft_consensus.cc:2466] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
W20250905 20:00:15.029697 30976 log.cc:927] Time spent T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.512s user 0.007s sys 0.002s
W20250905 20:00:15.034260 30823 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.020s user 0.031s sys 0.077s
I20250905 20:00:15.061555 28080 raft_consensus.cc:3058] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:15.061635 28123 leader_election.cc:304] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:15.032169 30885 raft_consensus.cc:491] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:15.066901 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:15.049758 30877 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:14.897655 30839 consensus_queue.cc:237] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:15.069736 27973 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:14.916280 31002 raft_consensus.cc:513] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:15.047360 30537 raft_consensus.cc:491] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:15.075793 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:15.081207 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 280001us
I20250905 20:00:15.081760 28077 raft_consensus.cc:2466] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:15.082995 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:15.083691 28077 raft_consensus.cc:3058] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:15.083796 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:15.080467 27973 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:15.084777 28082 raft_consensus.cc:2466] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 20:00:15.086021 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:15.087980 28082 raft_consensus.cc:2391] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 6: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:15.083247 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:15.091722 28150 raft_consensus.cc:3058] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:15.093545 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:15.062319 30939 raft_consensus.cc:491] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:15.108455 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:15.069491 30869 raft_consensus.cc:2802] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250905 20:00:15.080432 30775 consensus_meta.cc:287] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.686s user 0.012s sys 0.000s
I20250905 20:00:15.110592 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:15.134259 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:15.068743 30977 raft_consensus.cc:491] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:15.136188 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:15.137682 28082 raft_consensus.cc:2466] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 20:00:15.079411 30984 leader_election.cc:290] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:15.140838 28078 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:15.150899 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:15.145710 28077 raft_consensus.cc:2466] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6.
I20250905 20:00:15.152493 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:15.153360 28077 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6.
I20250905 20:00:15.160249 28077 raft_consensus.cc:1273] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20250905 20:00:15.161708 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 332785us
I20250905 20:00:15.146051 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:15.164785 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:15.165441 28151 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Advancing to term 9
I20250905 20:00:15.165839 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:15.139693 30992 log.cc:927] Time spent T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20250905 20:00:15.162052 28078 raft_consensus.cc:3058] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
W20250905 20:00:15.167956 27975 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:15.168285 28122 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.575s (SENT) [suppressed 35 similar messages]
W20250905 20:00:15.174222 27975 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:15.161818 28077 raft_consensus.cc:1238] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:15.176424 28077 raft_consensus.cc:1238] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:15.182413 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:15.185448 28077 raft_consensus.cc:3058] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:15.187198 28151 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 9.
I20250905 20:00:15.109007 30823 raft_consensus.cc:491] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:15.080225 30885 raft_consensus.cc:513] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:15.065891 30967 raft_consensus.cc:513] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:15.191450 28122 leader_election.cc:304] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:15.132680 30869 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:15.179847 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:15.193787 28081 raft_consensus.cc:2466] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
I20250905 20:00:15.223840 28155 raft_consensus.cc:2366] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 6. Current term is 7.
I20250905 20:00:15.103870 30537 raft_consensus.cc:513] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:15.210659 30823 raft_consensus.cc:513] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:15.187026 30835 leader_election.cc:290] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 20:00:15.216965 30986 log.cc:927] Time spent T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.992s user 0.001s sys 0.012s
I20250905 20:00:15.135078 30939 raft_consensus.cc:513] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:15.224436 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:15.198563 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:15.133451 30833 raft_consensus.cc:513] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:15.164258 30597 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:15.163755 30969 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:15.238972 30824 raft_consensus.cc:491] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:15.246407 28081 raft_consensus.cc:2466] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 20:00:15.243605 28153 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1660) took 1266 ms. Trace:
I20250905 20:00:15.147475 30775 raft_consensus.cc:513] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:15.251108 28008 raft_consensus.cc:1238] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250905 20:00:15.272735 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:15.260980 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:15.246378 30976 log.cc:927] Time spent T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20250905 20:00:15.230940 30869 raft_consensus.cc:3058] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:15.162194 30977 raft_consensus.cc:513] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:15.270191 28078 raft_consensus.cc:2466] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6.
I20250905 20:00:15.276252 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:15.291903 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 100576us
I20250905 20:00:15.269405 28153 rpcz_store.cc:276] 0905 20:00:13.977362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:13.980501 (+ 3139us) service_pool.cc:225] Handling call
0905 20:00:13.980973 (+ 472us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:13.987426 (+ 6453us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:13.987464 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:13.987954 (+ 490us) log.cc:844] Serialized 26 byte log entry
0905 20:00:14.125790 (+137836us) raft_consensus.cc:1643] Marking committed up to 3
0905 20:00:14.175840 (+ 50050us) log.cc:844] Serialized 14 byte log entry
0905 20:00:14.176132 (+ 292us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:14.176235 (+ 103us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:15.243264 (+1067029us) raft_consensus.cc:1683] finished
0905 20:00:15.243293 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:15.243495 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2259,"prepare.queue_time_us":145923,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"raft.queue_time_us":6470,"raft.run_cpu_time_us":350,"raft.run_wall_time_us":29772,"thread_start_us":100062,"threads_started":1,"wal-append.queue_time_us":646789}
I20250905 20:00:15.309506 28122 leader_election.cc:304] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:15.260187 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:15.278846 28077 raft_consensus.cc:2466] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:15.310393 28151 raft_consensus.cc:2466] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 6.
I20250905 20:00:15.311851 30663 raft_consensus.cc:2747] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:15.312124 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:15.270589 30861 raft_consensus.cc:491] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:15.313313 30861 raft_consensus.cc:513] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:15.308593 31010 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:15.316668 30861 leader_election.cc:290] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:15.302994 28157 raft_consensus.cc:2466] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
I20250905 20:00:15.317037 28050 leader_election.cc:304] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:15.268597 30984 raft_consensus.cc:491] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:15.304618 28078 raft_consensus.cc:1273] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250905 20:00:15.305230 28003 raft_consensus.cc:1238] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20250905 20:00:15.246901 30877 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:15.309572 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:15.286881 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:15.325012 28082 raft_consensus.cc:2466] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 20:00:15.312790 28151 raft_consensus.cc:2466] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
I20250905 20:00:15.313385 28122 leader_election.cc:304] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:15.303483 28081 raft_consensus.cc:1273] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20250905 20:00:15.273876 30668 raft_consensus.cc:2802] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 20:00:15.318214 28050 leader_election.cc:304] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:15.275861 31011 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:15.342602 28050 leader_election.cc:304] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:15.337152 28151 raft_consensus.cc:1273] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20250905 20:00:15.335096 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:15.302126 30933 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.402s user 0.037s sys 0.067s
I20250905 20:00:15.330732 28122 leader_election.cc:304] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
W20250905 20:00:15.343828 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:15.346051 28153 raft_consensus.cc:1273] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250905 20:00:15.377985 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:15.378669 28004 raft_consensus.cc:3058] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:15.379500 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:15.380230 28005 raft_consensus.cc:3058] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
W20250905 20:00:15.379155 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 214461us
I20250905 20:00:15.274173 30824 raft_consensus.cc:513] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:15.270535 30969 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:15.274046 30597 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:15.377112 30976 log.cc:927] Time spent T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.071s user 0.001s sys 0.000s
I20250905 20:00:15.302883 30938 consensus_queue.cc:1035] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.075s
I20250905 20:00:15.329192 30883 raft_consensus.cc:2802] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:15.376935 31004 raft_consensus.cc:2802] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:15.346648 28156 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39586 (request call id 356) took 1529 ms. Trace:
I20250905 20:00:15.341565 30668 raft_consensus.cc:491] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:15.386968 30660 raft_consensus.cc:2802] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:15.389716 30926 raft_consensus.cc:3058] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:15.475499 30668 raft_consensus.cc:3058] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:15.452729 30883 raft_consensus.cc:491] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:15.494709 30883 raft_consensus.cc:3058] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:15.456035 30975 consensus_queue.cc:1035] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20250905 20:00:15.314962 31010 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:15.462776 28156 rpcz_store.cc:276] 0905 20:00:13.817030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:13.817164 (+ 134us) service_pool.cc:225] Handling call
0905 20:00:13.831007 (+ 13843us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:13.854407 (+ 23400us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:13.854479 (+ 72us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:13.903431 (+ 48952us) log.cc:844] Serialized 26 byte log entry
0905 20:00:14.135254 (+231823us) raft_consensus.cc:1643] Marking committed up to 8
0905 20:00:14.136937 (+ 1683us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:14.137031 (+ 94us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:15.216162 (+1079131us) spinlock_profiling.cc:243] Waited 1.53 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49d85891e 00007fe49d857c66 00007fe49d857888 00007fe49d8612c1 00007fe49d861279 00007fe49d861209
0905 20:00:15.337784 (+121622us) raft_consensus.cc:1683] finished
0905 20:00:15.337816 (+ 32us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:15.346574 (+ 8758us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20115,"raft.queue_time_us":13017,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":9133,"spinlock_wait_cycles":3528064,"thread_start_us":187587,"threads_started":1,"wal-append.queue_time_us":272411}
I20250905 20:00:15.489045 28080 raft_consensus.cc:2466] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 7.
I20250905 20:00:15.490175 30937 raft_consensus.cc:491] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
W20250905 20:00:15.490924 31005 log.cc:927] Time spent T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.324s user 0.001s sys 0.000s
I20250905 20:00:15.492803 30969 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:15.337961 30984 raft_consensus.cc:513] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:15.437685 30773 consensus_queue.cc:1035] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.061s
I20250905 20:00:15.499053 30660 raft_consensus.cc:491] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
W20250905 20:00:15.348089 30933 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.448s user 0.037s sys 0.067s
I20250905 20:00:15.345225 30877 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:15.509394 28004 raft_consensus.cc:2466] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:15.574810 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:15.509692 28005 raft_consensus.cc:2408] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250905 20:00:15.506927 28084 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1624) took 1828 ms. Trace:
I20250905 20:00:15.376892 31011 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:15.462450 31004 raft_consensus.cc:491] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:15.527312 28007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 361) took 1505 ms. Trace:
I20250905 20:00:15.582906 28007 rpcz_store.cc:276] 0905 20:00:14.022138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:14.022297 (+ 159us) service_pool.cc:225] Handling call
0905 20:00:15.527118 (+1504821us) spinlock_profiling.cc:243] Waited 358 ms on lock 0x7b540034bf54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:15.527245 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":823088512}
I20250905 20:00:15.527534 30597 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:15.585968 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 194519us
I20250905 20:00:15.530721 30927 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:15.531715 31013 log.cc:927] Time spent T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.202s user 0.001s sys 0.000s
I20250905 20:00:15.532799 30968 raft_consensus.cc:491] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:15.534921 30925 raft_consensus.cc:491] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:15.535283 30537 leader_election.cc:290] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:15.585441 31004 raft_consensus.cc:3058] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:15.595775 30927 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:15.604974 30927 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:15.610210 30925 raft_consensus.cc:513] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:15.612670 30925 leader_election.cc:290] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:15.612886 28002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1522) took 2525 ms. Trace:
I20250905 20:00:15.517729 30937 raft_consensus.cc:513] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:15.539230 30984 leader_election.cc:290] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:15.541385 27975 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:15.601446 30968 raft_consensus.cc:513] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:15.617914 30877 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:15.543418 30823 leader_election.cc:290] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:15.546870 31002 leader_election.cc:290] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:15.551580 30967 leader_election.cc:290] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:15.556397 31022 raft_consensus.cc:491] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:15.619757 30968 leader_election.cc:290] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:15.619446 31022 raft_consensus.cc:513] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:15.620944 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 20:00:15.563598 30871 raft_consensus.cc:491] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:15.621932 30871 raft_consensus.cc:513] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:15.623739 28004 raft_consensus.cc:3058] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
W20250905 20:00:15.564977 31019 log.cc:927] Time spent T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
W20250905 20:00:15.567018 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 236360us
I20250905 20:00:15.568868 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:15.501956 31021 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:15.575919 28084 rpcz_store.cc:276] 0905 20:00:13.678657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:13.679570 (+ 913us) service_pool.cc:225] Handling call
0905 20:00:13.687054 (+ 7484us) raft_consensus.cc:1469] Updating replica for 2 ops
0905 20:00:13.738636 (+ 51582us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:13.738701 (+ 65us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0905 20:00:13.775832 (+ 37131us) log.cc:844] Serialized 52 byte log entry
0905 20:00:14.011028 (+235196us) raft_consensus.cc:1643] Marking committed up to 8
0905 20:00:14.092600 (+ 81572us) log.cc:844] Serialized 14 byte log entry
0905 20:00:14.092986 (+ 386us) log.cc:844] Serialized 14 byte log entry
0905 20:00:14.093149 (+ 163us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:14.093229 (+ 80us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:15.028923 (+935694us) spinlock_profiling.cc:243] Waited 26.4 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49d85891e 00007fe49d857c66 00007fe49d857888 00007fe49d8612c1 00007fe49d861279 00007fe49d861209
0905 20:00:15.506560 (+477637us) raft_consensus.cc:1683] finished
0905 20:00:15.506598 (+ 38us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:15.506829 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5822,"prepare.queue_time_us":131557,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":41,"raft.queue_time_us":75618,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":14269,"spinlock_wait_cycles":166836736,"thread_start_us":193761,"threads_started":1,"wal-append.queue_time_us":651054}
I20250905 20:00:15.580981 31032 consensus_queue.cc:1035] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 20:00:15.585527 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:15.653426 28005 raft_consensus.cc:2466] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
W20250905 20:00:15.653970 31018 log.cc:927] Time spent T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20250905 20:00:15.654606 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:15.529049 30969 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:15.657066 30969 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:15.601807 30885 leader_election.cc:290] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:15.573777 30660 raft_consensus.cc:3058] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:15.600198 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:15.622274 28158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2208) took 1483 ms. Trace:
I20250905 20:00:15.613373 28002 rpcz_store.cc:276] 0905 20:00:13.087620 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:13.087840 (+ 220us) service_pool.cc:225] Handling call
0905 20:00:13.088193 (+ 353us) raft_consensus.cc:1469] Updating replica for 2 ops
0905 20:00:13.358657 (+270464us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:13.358689 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0905 20:00:13.400069 (+ 41380us) log.cc:844] Serialized 52 byte log entry
0905 20:00:13.563776 (+163707us) raft_consensus.cc:1643] Marking committed up to 3
0905 20:00:13.772013 (+208237us) log.cc:844] Serialized 14 byte log entry
0905 20:00:13.950873 (+178860us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:15.612421 (+1661548us) spinlock_profiling.cc:243] Waited 270 ms on lock 0x7b540034c454. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:15.612480 (+ 59us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:15.612551 (+ 71us) raft_consensus.cc:1683] finished
0905 20:00:15.612653 (+ 102us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:15.612820 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":123990,"prepare.queue_time_us":223526,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":9452,"raft.queue_time_us":183488,"raft.run_cpu_time_us":318,"raft.run_wall_time_us":8987,"spinlock_wait_cycles":620943744,"thread_start_us":390780,"threads_started":3,"wal-append.queue_time_us":183569,"wal-append.run_cpu_time_us":2534,"wal-append.run_wall_time_us":1162139}
W20250905 20:00:15.629103 30973 log.cc:927] Time spent T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 1.750s user 0.005s sys 0.019s
I20250905 20:00:15.630757 30950 raft_consensus.cc:3053] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 LEADER]: Stepping down as leader of term 6
I20250905 20:00:15.631178 28010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 356) took 1938 ms. Trace:
W20250905 20:00:15.637300 30983 log.cc:927] Time spent T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 1.413s user 0.010s sys 0.007s
I20250905 20:00:15.639816 28122 leader_election.cc:304] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
W20250905 20:00:15.626260 28150 consensus_meta.cc:287] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.534s user 0.008s sys 0.003s
I20250905 20:00:15.638103 30741 raft_consensus.cc:3053] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 7 LEADER]: Stepping down as leader of term 7
W20250905 20:00:15.648581 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 107233us
I20250905 20:00:15.650411 30960 raft_consensus.cc:2802] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Leader election won for term 9
I20250905 20:00:15.598208 31014 raft_consensus.cc:491] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:15.598615 30833 leader_election.cc:290] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:15.654732 31021 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:15.996228 28008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1539) took 2000 ms. Trace:
W20250905 20:00:15.991978 28152 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.549s user 0.009s sys 0.051s
I20250905 20:00:15.772930 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:15.999996 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 60284.4us
I20250905 20:00:15.667945 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:15.998386 28150 raft_consensus.cc:2466] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 20:00:15.678081 28159 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1662) took 1700 ms. Trace:
I20250905 20:00:15.662135 28004 raft_consensus.cc:2466] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
W20250905 20:00:15.854933 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 468146us
W20250905 20:00:16.010491 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 370754us
W20250905 20:00:15.920454 27976 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.521s user 0.028s sys 0.033s
W20250905 20:00:16.011440 27976 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.612s user 0.030s sys 0.035s
I20250905 20:00:16.001565 28076 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1588) took 1011 ms. Trace:
I20250905 20:00:15.998234 28008 rpcz_store.cc:276] 0905 20:00:13.995829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:13.996136 (+ 307us) service_pool.cc:225] Handling call
0905 20:00:13.996442 (+ 306us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:15.275693 (+1279251us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:15.993361 (+717668us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b5400384ad4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:15.996127 (+ 2766us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2884505984}
W20250905 20:00:15.738062 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:15.793813 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 143715us
W20250905 20:00:16.007261 28152 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.615s user 0.011s sys 0.052s
I20250905 20:00:16.008847 28010 rpcz_store.cc:276] 0905 20:00:13.692647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:13.693275 (+ 628us) service_pool.cc:225] Handling call
0905 20:00:13.734933 (+ 41658us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:14.735042 (+1000109us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:15.630870 (+895828us) spinlock_profiling.cc:243] Waited 999 ms on lock 0x7b540016dfd4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:15.631106 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2297452800}
I20250905 20:00:15.934779 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:15.997915 28158 rpcz_store.cc:276] 0905 20:00:14.139121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:14.279812 (+140691us) service_pool.cc:225] Handling call
0905 20:00:14.445068 (+165256us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:14.832501 (+387433us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:14.832552 (+ 51us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:14.832617 (+ 65us) raft_consensus.cc:1643] Marking committed up to 7
0905 20:00:14.832663 (+ 46us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:15.622031 (+789368us) spinlock_profiling.cc:243] Waited 387 ms on lock 0x7b540009e7d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:15.622075 (+ 44us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:15.622196 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":890248960}
I20250905 20:00:16.007747 28007 raft_consensus.cc:2466] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 20:00:16.018682 28715 leader_election.cc:304] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:15.625943 30775 leader_election.cc:290] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 9 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:16.022760 28717 leader_election.cc:304] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:16.111071 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 20:00:16.111822 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 89171us
I20250905 20:00:16.112280 31031 raft_consensus.cc:2802] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader election won for term 5
I20250905 20:00:16.111091 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:16.010154 28159 rpcz_store.cc:276] 0905 20:00:13.978041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:13.981850 (+ 3809us) service_pool.cc:225] Handling call
0905 20:00:13.982252 (+ 402us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:13.982584 (+ 332us) raft_consensus.cc:1532] Early marking committed up to index 4
0905 20:00:13.994828 (+ 12244us) log.cc:844] Serialized 14 byte log entry
0905 20:00:15.669916 (+1675088us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b1800300160. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49d86555c 00007fe49ac873cb 00007fe4a38dcb99 00007fe49aca369b 00007fe49aca193f 00007fe49aca3c02 00007fe49d8572bb 00007fe49d85cdc3 00007fe49d85d2da
0905 20:00:15.670168 (+ 252us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:15.670231 (+ 63us) raft_consensus.cc:1643] Marking committed up to 4
0905 20:00:15.670291 (+ 60us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:15.670403 (+ 112us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:15.677979 (+ 7576us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":37712,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":59,"raft.queue_time_us":1932,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":123,"spinlock_wait_cycles":27680640,"thread_start_us":108731,"threads_started":1,"wal-append.queue_time_us":181345,"wal-append.run_cpu_time_us":5387,"wal-append.run_wall_time_us":1437994}
I20250905 20:00:16.048251 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:16.121084 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 98441.4us
I20250905 20:00:16.121440 28008 raft_consensus.cc:2466] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 6.
I20250905 20:00:16.015780 28001 raft_consensus.cc:2466] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 9.
I20250905 20:00:16.124927 28001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45844 (request call id 360) took 2125 ms. Trace:
I20250905 20:00:16.128890 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:16.115375 28155 raft_consensus.cc:1238] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 7. Current term is 8. Ops: []
I20250905 20:00:16.070101 28122 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:16.130632 30938 raft_consensus.cc:491] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:16.062080 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:16.131646 28080 raft_consensus.cc:1273] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250905 20:00:16.126164 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:16.132457 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:16.132723 28158 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 20:00:16.009145 28005 raft_consensus.cc:2466] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
W20250905 20:00:16.132887 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 114351us
I20250905 20:00:16.113358 30690 raft_consensus.cc:491] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:16.137475 30690 raft_consensus.cc:513] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:16.139849 30690 leader_election.cc:290] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:16.126536 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:16.141810 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:16.143275 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:16.113106 30537 consensus_queue.cc:1046] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20250905 20:00:16.143903 30969 leader_election.cc:290] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:16.144932 28078 raft_consensus.cc:1238] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:16.146687 30894 raft_consensus.cc:2802] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Leader election won for term 9
I20250905 20:00:16.147018 31021 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:16.147619 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:16.148015 31022 leader_election.cc:290] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:16.148424 28082 raft_consensus.cc:3058] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:16.149703 30888 raft_consensus.cc:2747] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 20:00:16.149925 30612 raft_consensus.cc:3053] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 LEADER]: Stepping down as leader of term 5
I20250905 20:00:16.126885 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:16.126232 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:16.151192 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 43875.3us
I20250905 20:00:16.151623 28003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1540) took 2155 ms. Trace:
I20250905 20:00:16.151822 28010 raft_consensus.cc:2466] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 6.
I20250905 20:00:16.151991 28003 rpcz_store.cc:276] 0905 20:00:13.996239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:13.996612 (+ 373us) service_pool.cc:225] Handling call
0905 20:00:14.011628 (+ 15016us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:15.336023 (+1324395us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:16.151333 (+815310us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b5400340dd4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:16.151556 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2974254976}
I20250905 20:00:16.128224 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:16.130738 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:16.112902 31031 raft_consensus.cc:695] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 5 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:16.160851 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:16.162070 28079 raft_consensus.cc:2391] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 6: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:16.162639 30537 raft_consensus.cc:3053] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Stepping down as leader of term 4
I20250905 20:00:16.163079 30537 raft_consensus.cc:738] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:16.160844 31031 consensus_queue.cc:237] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:16.163929 30537 consensus_queue.cc:260] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:16.165239 30537 raft_consensus.cc:3058] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
W20250905 20:00:15.901340 30944 log.cc:927] Time spent T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 2.631s user 0.000s sys 0.013s
W20250905 20:00:15.929903 30869 consensus_meta.cc:287] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.608s user 0.008s sys 0.003s
I20250905 20:00:16.049144 27975 leader_election.cc:400] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705) with higher term. Message: Invalid argument: T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 6. Current term is 7.
I20250905 20:00:16.129015 28001 rpcz_store.cc:276] 0905 20:00:13.999708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:14.000598 (+ 890us) service_pool.cc:225] Handling call
0905 20:00:15.772723 (+1772125us) spinlock_profiling.cc:243] Waited 4.12 ms on lock 0x7b5800000f98. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b7db60 00007fe4a2b7d444 00007fe49c2050c4 00007fe49c205052
0905 20:00:16.124869 (+352146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9481088}
I20250905 20:00:16.188077 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:16.188808 27975 leader_election.cc:403] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20250905 20:00:15.979138 30871 leader_election.cc:290] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:16.130986 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 61020us
W20250905 20:00:16.191418 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 142482us
I20250905 20:00:15.998270 30741 raft_consensus.cc:738] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:15.780799 30672 raft_consensus.cc:491] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:16.139701 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:16.154417 28008 raft_consensus.cc:3058] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:15.729110 30777 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:16.155670 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:16.157605 28007 raft_consensus.cc:2466] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 6.
I20250905 20:00:16.218945 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:15.815635 30988 log.cc:927] Time spent T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 1.289s user 0.007s sys 0.003s
W20250905 20:00:15.875183 30839 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.006s sys 0.008s
I20250905 20:00:15.662876 30756 raft_consensus.cc:3053] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Stepping down as leader of term 5
I20250905 20:00:16.159660 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:16.167299 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:16.133664 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:15.667892 30939 leader_election.cc:290] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:16.169185 30902 raft_consensus.cc:491] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:16.169879 30946 leader_election.cc:290] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:15.992163 30883 raft_consensus.cc:513] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:16.172610 28079 raft_consensus.cc:1238] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
W20250905 20:00:16.173105 30668 consensus_meta.cc:287] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.683s user 0.009s sys 0.000s
W20250905 20:00:16.048887 30931 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.001s sys 0.013s
I20250905 20:00:16.106962 30663 raft_consensus.cc:2747] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:16.120878 30960 raft_consensus.cc:695] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:15.931340 30937 leader_election.cc:290] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:15.808115 30926 raft_consensus.cc:2747] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:15.661139 30824 leader_election.cc:290] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:15.929253 30875 raft_consensus.cc:3053] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Stepping down as leader of term 4
I20250905 20:00:16.181713 30906 raft_consensus.cc:491] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:16.182098 28158 raft_consensus.cc:1273] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20250905 20:00:16.181746 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:16.182695 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:16.131403 30938 raft_consensus.cc:513] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:16.147338 30894 raft_consensus.cc:695] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:15.997046 30950 raft_consensus.cc:738] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:16.189072 28078 raft_consensus.cc:2466] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:15.661296 30977 leader_election.cc:290] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:16.190372 28082 raft_consensus.cc:2466] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 7.
I20250905 20:00:16.190480 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:16.133677 28715 leader_election.cc:304] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:16.296505 28008 raft_consensus.cc:2466] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
I20250905 20:00:15.660213 31011 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:16.012166 28076 rpcz_store.cc:276] 0905 20:00:14.990002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:15.010678 (+ 20676us) service_pool.cc:225] Handling call
0905 20:00:15.011015 (+ 337us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:15.011258 (+ 243us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:15.011282 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:15.011778 (+ 496us) log.cc:844] Serialized 26 byte log entry
0905 20:00:15.339240 (+327462us) raft_consensus.cc:1643] Marking committed up to 3
0905 20:00:15.339289 (+ 49us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:15.339358 (+ 69us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:15.666123 (+326765us) raft_consensus.cc:1683] finished
0905 20:00:15.666154 (+ 31us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:16.001469 (+335315us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":326296,"threads_started":1,"wal-append.queue_time_us":455870}
I20250905 20:00:16.297538 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:16.048406 31029 raft_consensus.cc:2747] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:16.297919 30932 raft_consensus.cc:491] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:16.194176 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:16.303025 28081 raft_consensus.cc:2466] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
I20250905 20:00:16.194957 28122 leader_election.cc:304] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:16.303620 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:16.304369 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 109509us
I20250905 20:00:16.305078 28122 leader_election.cc:304] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:16.305128 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:16.306660 28122 leader_election.cc:304] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:16.150380 30612 raft_consensus.cc:738] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:16.110098 30885 consensus_queue.cc:1035] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.069s
I20250905 20:00:16.212396 30982 raft_consensus.cc:3058] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 7
I20250905 20:00:16.215265 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:16.227527 28123 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.813s user 0.009s sys 0.017s
I20250905 20:00:16.231518 28159 raft_consensus.cc:3058] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Advancing to term 7
W20250905 20:00:16.233386 27974 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.701s user 0.010s sys 0.030s
I20250905 20:00:16.235280 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:16.170452 28085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1620) took 2495 ms. Trace:
W20250905 20:00:16.317144 27974 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.785s user 0.010s sys 0.030s
I20250905 20:00:16.237486 28125 leader_election.cc:304] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:16.317625 28001 raft_consensus.cc:2408] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20250905 20:00:16.318033 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 80639.2us
I20250905 20:00:16.318823 28125 leader_election.cc:304] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:16.237675 28080 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Advancing to term 9
I20250905 20:00:16.321441 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:16.237916 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:16.323166 28153 raft_consensus.cc:2391] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 7: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:16.238458 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:16.107820 31014 raft_consensus.cc:513] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:15.992636 31004 raft_consensus.cc:513] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:16.249305 28051 leader_election.cc:304] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:16.247231 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:16.302521 30824 raft_consensus.cc:2747] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250905 20:00:16.317380 28085 rpcz_store.cc:276] 0905 20:00:13.674988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:13.675181 (+ 193us) service_pool.cc:225] Handling call
0905 20:00:13.675586 (+ 405us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:13.701609 (+ 26023us) raft_consensus.cc:1532] Early marking committed up to index 1
0905 20:00:13.701659 (+ 50us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:13.779250 (+ 77591us) log.cc:844] Serialized 26 byte log entry
0905 20:00:13.994618 (+215368us) raft_consensus.cc:1643] Marking committed up to 1
0905 20:00:13.995379 (+ 761us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:14.010671 (+ 15292us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:15.636662 (+1625991us) spinlock_profiling.cc:243] Waited 1.63 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49d85891e 00007fe49d857c66 00007fe49d857888 00007fe49d8612c1 00007fe49d861279 00007fe49d861209
0905 20:00:16.170100 (+533438us) raft_consensus.cc:1683] finished
0905 20:00:16.170130 (+ 30us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:16.170352 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15436,"raft.queue_time_us":23320,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":36900,"spinlock_wait_cycles":3750656,"thread_start_us":158918,"threads_started":1,"wal-append.queue_time_us":346508}
I20250905 20:00:16.194109 30775 raft_consensus.cc:491] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:16.236797 30839 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.996s user 0.014s sys 0.008s
I20250905 20:00:16.336714 28076 raft_consensus.cc:2391] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 5: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:16.320597 28718 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
W20250905 20:00:16.338747 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 89575.5us
I20250905 20:00:16.264111 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:16.339870 28051 leader_election.cc:304] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:16.270920 28006 raft_consensus.cc:1238] T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 LEADER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20250905 20:00:16.217533 30777 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:16.217727 30672 raft_consensus.cc:513] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:16.256769 30902 raft_consensus.cc:513] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:16.316547 28002 raft_consensus.cc:2374] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 6. Re-sending same reply.
I20250905 20:00:16.357203 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:16.257964 30931 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.733s user 0.005s sys 0.013s
I20250905 20:00:16.299777 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:16.361557 28152 raft_consensus.cc:2391] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 8: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
W20250905 20:00:16.363024 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 44302.5us
I20250905 20:00:16.312021 31029 raft_consensus.cc:2747] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250905 20:00:16.363729 31025 raft_consensus.cc:491] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:16.363806 30849 raft_consensus.cc:491] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:16.272352 30741 consensus_queue.cc:260] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:16.297362 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 163750us
I20250905 20:00:16.377841 27973 leader_election.cc:304] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:16.391067 28715 leader_election.cc:304] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:16.392096 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:16.392804 28749 raft_consensus.cc:2466] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
I20250905 20:00:16.308450 28122 leader_election.cc:304] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:16.392926 28715 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:16.309513 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:16.311276 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:16.313445 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:16.316505 28123 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.902s user 0.009s sys 0.017s
I20250905 20:00:16.322953 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:16.394800 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 86479.8us
I20250905 20:00:16.395385 28082 raft_consensus.cc:2466] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 9.
I20250905 20:00:16.315274 30926 raft_consensus.cc:2802] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:16.162158 30971 consensus_queue.cc:1035] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.051s
I20250905 20:00:16.264668 30668 raft_consensus.cc:513] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:16.297360 30906 raft_consensus.cc:513] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:16.242841 30756 raft_consensus.cc:738] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:16.278856 30875 raft_consensus.cc:738] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:16.415048 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:16.415854 28150 raft_consensus.cc:1238] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 3. Current term is 4. Ops: []
I20250905 20:00:16.341225 28051 leader_election.cc:304] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
W20250905 20:00:16.417094 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 75945.2us
I20250905 20:00:16.196048 30869 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:16.423589 28157 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.832s user 0.056s sys 0.092s
I20250905 20:00:16.386008 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:16.358127 28002 raft_consensus.cc:2466] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:16.371699 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:16.425004 28157 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.834s user 0.056s sys 0.093s
I20250905 20:00:16.396943 28083 raft_consensus.cc:2466] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
I20250905 20:00:16.399001 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:16.337225 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:16.399821 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 20:00:16.241842 31020 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.705s user 0.011s sys 0.020s
I20250905 20:00:16.396941 28715 leader_election.cc:304] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:16.376216 31004 leader_election.cc:290] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:16.405406 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:16.408236 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:16.433669 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:16.400228 28084 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Advancing to term 9
I20250905 20:00:16.373941 30849 raft_consensus.cc:513] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:16.404646 30956 raft_consensus.cc:491] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:16.404750 30888 raft_consensus.cc:2802] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader election won for term 6
I20250905 20:00:16.410986 30597 raft_consensus.cc:2802] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:16.411296 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:16.411550 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:16.376483 30977 consensus_queue.cc:1035] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20250905 20:00:16.413722 30668 leader_election.cc:290] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:16.413997 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:16.420395 28747 raft_consensus.cc:2391] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 6: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 20:00:16.371910 31029 raft_consensus.cc:2802] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 20:00:16.363675 31011 raft_consensus.cc:2802] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250905 20:00:16.417852 28051 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:16.460124 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:16.460835 28051 leader_election.cc:304] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:16.381625 30975 raft_consensus.cc:2747] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:16.390111 30985 raft_consensus.cc:2747] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:16.312582 30946 raft_consensus.cc:2747] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 20:00:16.426813 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:16.373227 30672 leader_election.cc:290] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:16.383242 31041 raft_consensus.cc:491] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:16.315048 30932 raft_consensus.cc:513] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:16.378935 30773 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:16.389508 30950 consensus_queue.cc:260] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:16.368247 31043 raft_consensus.cc:2802] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election won for term 6
I20250905 20:00:16.430788 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:16.430645 28715 leader_election.cc:304] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:16.395042 31033 raft_consensus.cc:2747] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:16.432708 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:16.459156 31011 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:16.404203 30926 raft_consensus.cc:491] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:16.414371 30741 raft_consensus.cc:3058] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:16.300015 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:16.530560 28008 raft_consensus.cc:2466] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 6.
I20250905 20:00:16.532088 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:16.532931 28008 raft_consensus.cc:3058] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:16.444593 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:16.523834 28005 raft_consensus.cc:2391] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:16.377029 30850 consensus_queue.cc:1046] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
W20250905 20:00:16.429416 31020 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.903s user 0.011s sys 0.020s
I20250905 20:00:16.382134 30775 raft_consensus.cc:513] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:16.453188 28750 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
W20250905 20:00:16.455081 28154 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.521s user 0.027s sys 0.105s
I20250905 20:00:16.372426 31025 raft_consensus.cc:513] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:16.429694 30937 raft_consensus.cc:2802] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:16.399279 30839 consensus_queue.cc:1046] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20250905 20:00:16.463297 27974 leader_election.cc:304] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:16.480660 28155 raft_consensus.cc:2391] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 6: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:16.440160 30824 raft_consensus.cc:2747] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250905 20:00:16.442925 30597 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:16.392715 27973 leader_election.cc:304] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 20:00:16.465794 30980 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.645s user 0.034s sys 0.042s
I20250905 20:00:16.502830 28084 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 9.
I20250905 20:00:16.500484 28080 raft_consensus.cc:2466] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 9.
W20250905 20:00:16.405635 31035 consensus_queue.cc:1162] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" responder_term: 8 status { last_received { term: 7 index: 8 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 7. Current term is 8. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250905 20:00:16.455505 30888 raft_consensus.cc:695] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 6 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:16.432813 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:16.462347 31029 raft_consensus.cc:491] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:16.626291 30850 raft_consensus.cc:2802] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:16.406483 30960 consensus_queue.cc:237] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:16.517268 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 86745.8us
I20250905 20:00:16.524757 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:16.665354 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:16.516129 28159 raft_consensus.cc:2466] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 7.
I20250905 20:00:16.518886 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:16.667335 28001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 2095) took 1142 ms. Trace:
I20250905 20:00:16.480434 28743 raft_consensus.cc:1238] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:16.547951 28005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 2093) took 1023 ms. Trace:
I20250905 20:00:16.483693 31004 raft_consensus.cc:2747] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:16.446542 30956 raft_consensus.cc:513] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:16.499184 30672 raft_consensus.cc:2747] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250905 20:00:16.584672 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:16.590590 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 173503us
I20250905 20:00:16.427541 30894 consensus_queue.cc:237] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:16.557054 28154 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.624s user 0.027s sys 0.105s
I20250905 20:00:16.558228 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:16.523046 30946 raft_consensus.cc:2802] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 20:00:16.530017 30926 raft_consensus.cc:3058] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:16.530166 30950 raft_consensus.cc:3058] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:16.514917 30883 leader_election.cc:290] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:16.524032 30975 raft_consensus.cc:491] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:16.662832 30850 raft_consensus.cc:491] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250905 20:00:16.724231 31044 log.cc:927] Time spent T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.123s user 0.000s sys 0.000s
I20250905 20:00:16.601745 27975 leader_election.cc:304] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
W20250905 20:00:16.756989 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 155311us
I20250905 20:00:16.758066 27975 leader_election.cc:304] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:16.756750 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:16.712443 30946 raft_consensus.cc:491] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250905 20:00:16.509826 31024 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.872s user 0.009s sys 0.016s
W20250905 20:00:16.612233 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 216119us
W20250905 20:00:16.614714 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 222037us
I20250905 20:00:16.733167 30850 raft_consensus.cc:3058] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:16.574911 30777 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:16.616088 28008 raft_consensus.cc:2466] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 7.
I20250905 20:00:16.797760 28008 raft_consensus.cc:1238] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20250905 20:00:16.621255 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:16.623994 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 160759us
I20250905 20:00:16.530555 30885 consensus_queue.cc:1035] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.074s
I20250905 20:00:16.514762 30773 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:16.617374 30980 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.817s user 0.034s sys 0.043s
W20250905 20:00:16.646577 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 275646us
I20250905 20:00:16.603236 30937 raft_consensus.cc:491] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:16.576109 31043 raft_consensus.cc:695] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:16.502279 31041 raft_consensus.cc:513] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:16.619731 30597 raft_consensus.cc:3058] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:16.600678 31020 raft_consensus.cc:3053] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 LEADER]: Stepping down as leader of term 6
I20250905 20:00:16.616662 30902 leader_election.cc:290] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:16.843997 31038 log.cc:927] Time spent T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.621s user 0.000s sys 0.002s
I20250905 20:00:16.638789 30927 raft_consensus.cc:491] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:16.845898 30862 raft_consensus.cc:491] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:16.845678 31043 consensus_queue.cc:237] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:16.846791 28157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2226) took 1261 ms. Trace:
I20250905 20:00:16.846745 30862 raft_consensus.cc:513] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:16.847366 28157 rpcz_store.cc:276] 0905 20:00:15.585002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:15.585609 (+ 607us) service_pool.cc:225] Handling call
0905 20:00:15.585948 (+ 339us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:16.427657 (+841709us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:16.427714 (+ 57us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:16.436399 (+ 8685us) log.cc:844] Serialized 26 byte log entry
0905 20:00:16.638557 (+202158us) raft_consensus.cc:1643] Marking committed up to 5
0905 20:00:16.638596 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:16.638680 (+ 84us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:16.846516 (+207836us) raft_consensus.cc:1683] finished
0905 20:00:16.846539 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:16.846733 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":52924,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":6140,"thread_start_us":1031072,"threads_started":2,"wal-append.queue_time_us":273347}
I20250905 20:00:16.849308 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:16.850032 28157 raft_consensus.cc:2391] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 9: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:16.642084 31000 raft_consensus.cc:2802] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:16.850968 31000 raft_consensus.cc:491] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:16.850809 30926 raft_consensus.cc:513] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:16.851389 31000 raft_consensus.cc:3058] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:16.667833 28001 rpcz_store.cc:276] 0905 20:00:15.525342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:16.463752 (+938410us) service_pool.cc:225] Handling call
0905 20:00:16.667254 (+203502us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:16.855075 28001 raft_consensus.cc:1238] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250905 20:00:16.816726 30937 raft_consensus.cc:3058] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:16.855971 28001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1545) took 1487 ms. Trace:
I20250905 20:00:16.856462 28001 rpcz_store.cc:276] 0905 20:00:15.368246 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:16.854242 (+1485996us) service_pool.cc:225] Handling call
0905 20:00:16.854762 (+ 520us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:16.855626 (+ 864us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:16.855883 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:16.668423 28005 rpcz_store.cc:276] 0905 20:00:15.524853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:16.431578 (+906725us) service_pool.cc:225] Handling call
0905 20:00:16.547861 (+116283us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2816}
I20250905 20:00:16.858724 31057 consensus_queue.cc:1046] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 20:00:16.640189 30938 leader_election.cc:290] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:16.859779 31057 raft_consensus.cc:3053] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 LEADER]: Stepping down as leader of term 6
I20250905 20:00:16.669234 28715 leader_election.cc:304] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:16.860225 31057 raft_consensus.cc:738] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
W20250905 20:00:16.860464 31036 log.cc:927] Time spent T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.636s user 0.002s sys 0.000s
W20250905 20:00:16.860942 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 191801us
I20250905 20:00:16.706825 28003 raft_consensus.cc:2391] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 7: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:16.662302 30906 leader_election.cc:290] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:16.601884 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 141141us
I20250905 20:00:16.869421 28051 leader_election.cc:304] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:16.870852 28051 leader_election.cc:304] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:16.668839 31029 raft_consensus.cc:3058] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:16.723835 31025 leader_election.cc:290] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 20:00:16.510401 30961 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.989s user 0.014s sys 0.021s
I20250905 20:00:16.722075 28010 raft_consensus.cc:1238] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
W20250905 20:00:16.717451 30660 consensus_meta.cc:287] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.665s user 0.012s sys 0.000s
W20250905 20:00:16.734416 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 95048.6us
I20250905 20:00:16.738751 30932 leader_election.cc:290] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:16.907058 28010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1543) took 1565 ms. Trace:
I20250905 20:00:16.861335 31020 raft_consensus.cc:738] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:16.731918 30849 leader_election.cc:290] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:16.721365 30888 consensus_queue.cc:237] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:16.747179 28009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 363) took 1414 ms. Trace:
I20250905 20:00:16.706550 30869 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:16.748420 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:16.923435 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:16.710565 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:16.923403 28009 rpcz_store.cc:276] 0905 20:00:15.332646 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:15.332800 (+ 154us) service_pool.cc:225] Handling call
0905 20:00:15.336215 (+ 3415us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:15.393197 (+ 56982us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:15.393244 (+ 47us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:15.393316 (+ 72us) raft_consensus.cc:1643] Marking committed up to 6
0905 20:00:15.393390 (+ 74us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:16.746797 (+1353407us) spinlock_profiling.cc:243] Waited 2.79 ms on lock 0x7b54001fb154. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:16.746869 (+ 72us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:16.747082 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46038,"raft.queue_time_us":176487,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":115,"spinlock_wait_cycles":6413696}
I20250905 20:00:16.924863 28159 raft_consensus.cc:2466] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
W20250905 20:00:16.897419 30961 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.376s user 0.014s sys 0.024s
I20250905 20:00:16.789958 27973 leader_election.cc:304] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:16.530866 31011 raft_consensus.cc:3058] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Advancing to term 10
W20250905 20:00:16.790660 28004 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1551) took 1404 ms (client timeout 1605 ms). Trace:
W20250905 20:00:16.930711 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 140814us
W20250905 20:00:16.931005 28004 rpcz_store.cc:269] 0905 20:00:15.386475 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:15.574521 (+188046us) service_pool.cc:225] Handling call
0905 20:00:16.758731 (+1184210us) spinlock_profiling.cc:243] Waited 48.1 ms on lock 0x7b54001fc554. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:16.790592 (+ 31861us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":110661376}
I20250905 20:00:16.931584 27973 leader_election.cc:304] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:16.800284 28152 raft_consensus.cc:2466] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:16.801524 28008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1544) took 1433 ms. Trace:
W20250905 20:00:16.792243 31024 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.154s user 0.009s sys 0.017s
I20250905 20:00:16.754482 31010 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:16.724633 30975 raft_consensus.cc:513] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:16.786391 30839 raft_consensus.cc:2747] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:16.816246 28125 leader_election.cc:304] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:16.755812 30971 raft_consensus.cc:2802] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader election won for term 6
I20250905 20:00:16.818114 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:16.793917 30946 raft_consensus.cc:3058] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:16.788671 31007 raft_consensus.cc:2747] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:16.823861 30775 leader_election.cc:290] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:16.831753 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:16.968065 28150 raft_consensus.cc:3058] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:16.840596 28154 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1656) took 3143 ms. Trace:
I20250905 20:00:16.970667 28154 rpcz_store.cc:276] 0905 20:00:13.696941 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:13.721288 (+ 24347us) service_pool.cc:225] Handling call
0905 20:00:13.721700 (+ 412us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:13.798732 (+ 77032us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:13.798756 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:13.814198 (+ 15442us) log.cc:844] Serialized 26 byte log entry
0905 20:00:15.933389 (+2119191us) spinlock_profiling.cc:243] Waited 1.73 ms on lock 0x7b18000fab40. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49ac86d47 00007fe4a38dcb99 00007fe49aca369b 00007fe49aca193f 00007fe49aca3c02 00007fe49d8572bb 00007fe49d85cdc3 00007fe49d85d171 00007fe49dbec09f
0905 20:00:16.705688 (+772299us) raft_consensus.cc:1643] Marking committed up to 2
0905 20:00:16.705740 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:16.705932 (+ 192us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:16.840260 (+134328us) raft_consensus.cc:1683] finished
0905 20:00:16.840281 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:16.840524 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39938,"raft.queue_time_us":43923,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":21976,"spinlock_wait_cycles":3990272,"thread_start_us":772027,"threads_started":1,"wal-append.queue_time_us":2838579}
I20250905 20:00:16.841009 31051 raft_consensus.cc:491] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:16.933007 30862 leader_election.cc:290] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:16.841598 31049 log.cc:927] Time spent T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.171s user 0.001s sys 0.000s
I20250905 20:00:16.421253 30612 consensus_queue.cc:260] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:16.955829 30684 raft_consensus.cc:2802] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:16.845789 30927 raft_consensus.cc:513] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:16.867728 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
W20250905 20:00:16.986075 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 186024us
I20250905 20:00:16.872102 28003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 2103) took 1344 ms. Trace:
I20250905 20:00:16.879134 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:16.889673 28051 leader_election.cc:304] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:16.901258 28750 raft_consensus.cc:3058] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:16.743927 30956 leader_election.cc:290] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:16.907527 28010 rpcz_store.cc:276] 0905 20:00:15.367440 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:16.709536 (+1342096us) service_pool.cc:225] Handling call
0905 20:00:16.710064 (+ 528us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:16.906717 (+196653us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:16.906992 (+ 275us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:16.891278 30926 leader_election.cc:290] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:16.867324 30974 raft_consensus.cc:2802] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader election won for term 7
I20250905 20:00:16.997745 28157 raft_consensus.cc:1238] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:16.926156 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:16.470989 30875 consensus_queue.cc:260] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:16.927979 28050 leader_election.cc:304] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:16.934581 28008 rpcz_store.cc:276] 0905 20:00:15.367933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:16.796919 (+1428986us) service_pool.cc:225] Handling call
0905 20:00:16.797373 (+ 454us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:16.798340 (+ 967us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:16.801429 (+ 3089us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:16.937870 27973 leader_election.cc:304] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
W20250905 20:00:17.004722 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 66941.9us
I20250905 20:00:17.005932 28157 raft_consensus.cc:1238] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:16.902606 30902 raft_consensus.cc:2747] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250905 20:00:16.917109 30537 consensus_meta.cc:287] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.738s user 0.013s sys 0.000s
I20250905 20:00:16.950917 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:16.952126 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:16.911123 31004 raft_consensus.cc:2802] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election won for term 6
I20250905 20:00:16.953261 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:17.016516 28152 raft_consensus.cc:2466] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:16.955852 28155 raft_consensus.cc:3058] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Advancing to term 7
W20250905 20:00:16.955056 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 138902us
I20250905 20:00:16.962186 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:17.021518 28125 leader_election.cc:304] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:16.939435 30849 raft_consensus.cc:2802] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader election won for term 7
I20250905 20:00:16.861111 31057 consensus_queue.cc:260] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:16.981992 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:16.982573 28004 raft_consensus.cc:1273] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250905 20:00:16.862084 28151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2225) took 1293 ms. Trace:
I20250905 20:00:16.911352 30660 raft_consensus.cc:513] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:16.985493 28150 raft_consensus.cc:2466] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 7.
W20250905 20:00:16.985814 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 196240us
I20250905 20:00:17.029342 28158 raft_consensus.cc:2408] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250905 20:00:17.030449 28006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1542) took 2844 ms. Trace:
I20250905 20:00:16.993098 30883 raft_consensus.cc:2747] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20250905 20:00:16.989157 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 99615.8us
I20250905 20:00:16.989602 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:17.027299 27975 leader_election.cc:304] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:16.991334 27974 leader_election.cc:304] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:16.998845 28159 raft_consensus.cc:2466] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
I20250905 20:00:17.037158 28051 leader_election.cc:304] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:16.974130 30971 raft_consensus.cc:695] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
W20250905 20:00:17.010612 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 82750.2us
I20250905 20:00:17.006414 27973 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:16.841313 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:16.988823 30684 raft_consensus.cc:491] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:17.024987 28154 raft_consensus.cc:2466] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
I20250905 20:00:17.030351 28122 leader_election.cc:304] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:16.963364 30869 raft_consensus.cc:2802] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:17.023178 28151 rpcz_store.cc:276] 0905 20:00:15.568212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:15.576501 (+ 8289us) service_pool.cc:225] Handling call
0905 20:00:15.576917 (+ 416us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:15.585188 (+ 8271us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:15.585224 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:15.592675 (+ 7451us) log.cc:844] Serialized 26 byte log entry
0905 20:00:16.625971 (+1033296us) spinlock_profiling.cc:243] Waited 4.58 ms on lock 0x7b1800351340. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49ac86d47 00007fe4a38dcb99 00007fe49aca369b 00007fe49aca193f 00007fe49aca3c02 00007fe49d8572bb 00007fe49d85cdc3 00007fe49d85d171 00007fe49dbec09f
0905 20:00:16.796045 (+170074us) raft_consensus.cc:1643] Marking committed up to 4
0905 20:00:16.796089 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:16.796176 (+ 87us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:16.854159 (+ 57983us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49d85891e 00007fe49d857c66 00007fe49d857888 00007fe49d8612c1 00007fe49d861279 00007fe49d861209
0905 20:00:16.861827 (+ 7668us) raft_consensus.cc:1683] finished
0905 20:00:16.861849 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:16.861974 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":38091,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":23893,"spinlock_wait_cycles":36005120,"thread_start_us":150917,"threads_started":1,"wal-append.queue_time_us":1226813}
I20250905 20:00:17.058780 28050 leader_election.cc:304] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:17.059820 28050 leader_election.cc:304] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:17.060177 28151 raft_consensus.cc:1238] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 7. Current term is 8. Ops: []
W20250905 20:00:17.060433 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 54102.6us
I20250905 20:00:16.988381 28003 rpcz_store.cc:276] 0905 20:00:15.527804 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:16.557814 (+1030010us) service_pool.cc:225] Handling call
0905 20:00:16.872002 (+314188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:17.033847 28157 raft_consensus.cc:1238] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20250905 20:00:17.031040 28006 rpcz_store.cc:276] 0905 20:00:14.185943 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:14.186155 (+ 212us) service_pool.cc:225] Handling call
0905 20:00:14.186624 (+ 469us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:16.340763 (+2154139us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:17.013050 (+672287us) spinlock_profiling.cc:243] Waited 2.08 s on lock 0x7b54000555d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:17.030357 (+ 17307us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4792445696}
I20250905 20:00:16.967571 31059 raft_consensus.cc:491] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:17.041514 28051 leader_election.cc:304] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:17.042805 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:16.995266 30612 raft_consensus.cc:3058] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
W20250905 20:00:17.025787 31054 log.cc:927] Time spent T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.138s user 0.000s sys 0.001s
I20250905 20:00:17.069664 27975 leader_election.cc:304] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 20:00:17.049839 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 58548.8us
I20250905 20:00:17.070487 28051 leader_election.cc:304] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:17.025485 30938 raft_consensus.cc:2802] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:17.072800 28051 leader_election.cc:304] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:17.016038 30974 raft_consensus.cc:695] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 7 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:16.976526 31020 consensus_queue.cc:260] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:17.088886 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 58625.8us
I20250905 20:00:17.097455 28155 raft_consensus.cc:2466] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 7.
I20250905 20:00:17.076606 28717 leader_election.cc:304] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:17.092039 28003 raft_consensus.cc:1273] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20250905 20:00:17.100401 28717 leader_election.cc:304] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:17.091621 27974 leader_election.cc:304] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:17.043133 31004 raft_consensus.cc:695] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [term 6 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:17.058681 30684 raft_consensus.cc:3058] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:17.066489 30869 raft_consensus.cc:491] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:17.047365 30849 raft_consensus.cc:695] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:17.042865 31061 raft_consensus.cc:2802] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:16.987318 31051 raft_consensus.cc:513] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:17.077716 30672 raft_consensus.cc:2802] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election won for term 5
I20250905 20:00:17.077346 30883 raft_consensus.cc:2802] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader election won for term 7
I20250905 20:00:17.069082 30675 raft_consensus.cc:491] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:17.021728 31060 consensus_queue.cc:1046] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.100s
I20250905 20:00:17.077252 30956 raft_consensus.cc:2802] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250905 20:00:17.068429 30148 raft_consensus.cc:2747] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250905 20:00:17.067492 30839 raft_consensus.cc:2802] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:17.078119 30773 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:17.077558 30902 raft_consensus.cc:2802] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 20:00:17.090924 31043 raft_consensus.cc:491] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:17.090251 31057 raft_consensus.cc:3058] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:17.066771 30985 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:17.117195 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:17.090370 30938 raft_consensus.cc:491] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:17.089890 30931 raft_consensus.cc:2747] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:17.047588 30537 consensus_queue.cc:1046] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.060s
W20250905 20:00:17.127254 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 54549us
I20250905 20:00:17.204818 28051 leader_election.cc:304] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:17.206085 28051 leader_election.cc:304] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:17.207279 28051 leader_election.cc:304] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:17.104370 30977 raft_consensus.cc:2802] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:17.110337 27973 leader_election.cc:304] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:17.107954 31025 raft_consensus.cc:3058] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:17.145205 30871 consensus_queue.cc:1046] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20250905 20:00:17.228878 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 128651us
I20250905 20:00:17.109539 30823 raft_consensus.cc:2802] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader election won for term 7
I20250905 20:00:17.103789 31064 raft_consensus.cc:2802] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader election won for term 7
W20250905 20:00:17.120030 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:17.077069 31023 consensus_queue.cc:1035] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20250905 20:00:17.204344 30975 leader_election.cc:290] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:17.168627 30675 raft_consensus.cc:513] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:17.222958 30977 raft_consensus.cc:491] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:17.116492 31061 raft_consensus.cc:491] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:17.104245 30840 consensus_queue.cc:1046] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.062s
I20250905 20:00:17.103667 30906 consensus_queue.cc:1046] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.063s
I20250905 20:00:17.185343 28748 raft_consensus.cc:2466] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 3.
W20250905 20:00:17.188839 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 129095us
I20250905 20:00:17.237836 30931 raft_consensus.cc:2762] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20250905 20:00:17.168565 30956 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:17.224349 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:17.225883 28051 leader_election.cc:304] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:17.090489 31059 raft_consensus.cc:513] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:17.251152 30823 raft_consensus.cc:695] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:17.174077 30839 raft_consensus.cc:491] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:17.187076 31043 raft_consensus.cc:513] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:17.232707 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 122441us
I20250905 20:00:17.177639 30902 raft_consensus.cc:491] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:17.186908 30773 raft_consensus.cc:2747] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250905 20:00:17.203723 31046 log.cc:927] Time spent T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20250905 20:00:17.123853 31067 consensus_queue.cc:1046] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.110s
I20250905 20:00:17.202744 30938 raft_consensus.cc:3058] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:17.245193 30537 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:17.234097 31041 leader_election.cc:290] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:17.117453 31020 raft_consensus.cc:3058] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Advancing to term 7
W20250905 20:00:17.250155 27975 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:17.107692 30926 consensus_queue.cc:1035] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20250905 20:00:17.118571 30869 raft_consensus.cc:3058] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:17.260797 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:17.241659 30978 raft_consensus.cc:2802] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Leader election won for term 9
W20250905 20:00:17.325825 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 100023us
I20250905 20:00:17.189199 31078 raft_consensus.cc:2747] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:17.329131 28051 leader_election.cc:304] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 20:00:17.329587 31073 log.cc:927] Time spent T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20250905 20:00:17.239466 30777 raft_consensus.cc:2747] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:17.240195 30672 raft_consensus.cc:695] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [term 5 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:17.272370 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 20:00:17.254853 30982 consensus_meta.cc:287] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.871s user 0.012s sys 0.004s
I20250905 20:00:17.146624 30588 raft_consensus.cc:2747] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250905 20:00:17.279611 28715 connection.cc:232] RPC call timeout handler was delayed by -2.1617796409999528s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
W20250905 20:00:17.266408 31086 log.cc:927] Time spent T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.070s user 0.000s sys 0.001s
I20250905 20:00:17.174232 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:17.251823 31064 raft_consensus.cc:695] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:17.292196 28750 raft_consensus.cc:1273] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20250905 20:00:17.183837 30971 consensus_queue.cc:237] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:17.262297 30977 raft_consensus.cc:3058] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Advancing to term 7
W20250905 20:00:17.295034 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:17.263881 30975 raft_consensus.cc:2747] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:17.221175 30985 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:17.267907 31061 raft_consensus.cc:3058] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
W20250905 20:00:17.310046 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 175187us
I20250905 20:00:17.277168 30956 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Advancing to term 10
I20250905 20:00:17.309815 30883 raft_consensus.cc:695] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [term 7 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:17.287412 30660 leader_election.cc:290] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:17.236240 31004 consensus_queue.cc:237] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 20:00:17.292863 30839 raft_consensus.cc:3058] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:17.297067 30902 raft_consensus.cc:3058] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:17.247534 30849 consensus_queue.cc:237] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:17.235797 30974 consensus_queue.cc:237] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:17.345428 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:17.311180 31092 raft_consensus.cc:491] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
W20250905 20:00:17.352470 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 83810.3us
I20250905 20:00:17.297818 31081 consensus_queue.cc:1035] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
W20250905 20:00:17.362547 31074 log.cc:927] Time spent T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.093s user 0.001s sys 0.000s
W20250905 20:00:17.368628 30741 consensus_meta.cc:287] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.817s user 0.009s sys 0.002s
W20250905 20:00:17.371429 31063 log.cc:927] Time spent T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.244s user 0.000s sys 0.001s
I20250905 20:00:17.343035 30982 raft_consensus.cc:2747] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705) with higher term. Message: Invalid argument: T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 6. Current term is 7.
I20250905 20:00:17.371224 30672 consensus_queue.cc:237] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:17.424731 28751 raft_consensus.cc:2466] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 8.
I20250905 20:00:17.372956 31043 leader_election.cc:290] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:17.377296 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 48256.1us
I20250905 20:00:17.431056 28051 leader_election.cc:304] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:17.431999 28007 raft_consensus.cc:2466] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 7.
I20250905 20:00:17.387117 28150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1684) took 1068 ms. Trace:
I20250905 20:00:17.315392 30537 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:17.327373 30773 consensus_queue.cc:1046] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.047s
I20250905 20:00:17.374226 30777 raft_consensus.cc:2747] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250905 20:00:17.407522 31072 log.cc:927] Time spent T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.180s user 0.000s sys 0.001s
W20250905 20:00:17.412490 31077 log.cc:927] Time spent T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20250905 20:00:17.388231 30823 consensus_queue.cc:237] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:17.413175 31064 consensus_queue.cc:237] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:17.421119 31069 log.cc:927] Time spent T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.269s user 0.000s sys 0.001s
I20250905 20:00:17.421327 30923 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:17.381443 31051 leader_election.cc:290] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:17.409070 28050 leader_election.cc:304] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 20:00:17.439141 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 164206us
I20250905 20:00:17.434875 28150 rpcz_store.cc:276] 0905 20:00:16.318256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:17.035145 (+716889us) service_pool.cc:225] Handling call
0905 20:00:17.035498 (+ 353us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:17.035699 (+ 201us) raft_consensus.cc:1532] Early marking committed up to index 3
0905 20:00:17.105956 (+ 70257us) log.cc:844] Serialized 14 byte log entry
0905 20:00:17.386719 (+280763us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:17.386780 (+ 61us) raft_consensus.cc:1643] Marking committed up to 3
0905 20:00:17.386838 (+ 58us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:17.386913 (+ 75us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:17.387056 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10470,"prepare.queue_time_us":102699,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":62,"raft.queue_time_us":52448,"thread_start_us":306591,"threads_started":2}
I20250905 20:00:17.468685 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:17.469394 28077 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 10.
I20250905 20:00:17.470953 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:17.471760 28077 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 20:00:17.471964 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 20:00:17.445828 31031 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.254s user 0.009s sys 0.017s
I20250905 20:00:17.473872 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:17.475279 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:17.444911 30950 consensus_meta.cc:287] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.723s user 0.006s sys 0.008s
I20250905 20:00:17.478696 28077 raft_consensus.cc:2466] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 20:00:17.481585 28084 raft_consensus.cc:1238] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:17.483482 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:17.444294 30931 raft_consensus.cc:2747] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250905 20:00:17.477972 31031 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.291s user 0.010s sys 0.017s
I20250905 20:00:17.396879 30883 consensus_queue.cc:237] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:17.487197 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 78229.3us
I20250905 20:00:17.492913 28715 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:17.497938 28715 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:17.500192 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:17.465656 30675 leader_election.cc:290] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:17.465234 30660 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
W20250905 20:00:17.497119 30597 consensus_meta.cc:287] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.662s user 0.009s sys 0.005s
I20250905 20:00:17.528602 28049 leader_election.cc:304] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:17.527261 30985 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:17.534520 28715 leader_election.cc:304] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:17.495738 30975 raft_consensus.cc:2802] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader election won for term 7
I20250905 20:00:17.547426 28001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1554) took 1261 ms. Trace:
I20250905 20:00:17.414110 31092 raft_consensus.cc:513] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:17.553100 28001 rpcz_store.cc:276] 0905 20:00:16.266949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:17.527276 (+1260327us) service_pool.cc:225] Handling call
0905 20:00:17.527629 (+ 353us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:17.527806 (+ 177us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:17.527833 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:17.527874 (+ 41us) raft_consensus.cc:1643] Marking committed up to 4
0905 20:00:17.527909 (+ 35us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:17.527965 (+ 56us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:17.528066 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:17.550197 28715 leader_election.cc:304] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:17.506230 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:17.556422 30975 raft_consensus.cc:695] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 7 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:17.556653 31023 raft_consensus.cc:2802] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:17.556465 31035 raft_consensus.cc:2747] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:17.558702 28009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 2113) took 2028 ms. Trace:
I20250905 20:00:17.524653 30741 raft_consensus.cc:2802] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250905 20:00:17.529465 31031 raft_consensus.cc:2802] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250905 20:00:17.551183 30946 consensus_meta.cc:287] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.577s user 0.007s sys 0.004s
W20250905 20:00:17.548795 30850 consensus_meta.cc:287] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.740s user 0.007s sys 0.011s
I20250905 20:00:17.571924 28009 rpcz_store.cc:276] 0905 20:00:15.530295 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:16.925501 (+1395206us) service_pool.cc:225] Handling call
0905 20:00:16.930504 (+ 5003us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:16.930872 (+ 368us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:16.930904 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:16.945612 (+ 14708us) log.cc:844] Serialized 26 byte log entry
0905 20:00:17.057341 (+111729us) raft_consensus.cc:1643] Marking committed up to 4
0905 20:00:17.187194 (+129853us) log.cc:844] Serialized 14 byte log entry
0905 20:00:17.196913 (+ 9719us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:17.197005 (+ 92us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:17.558476 (+361471us) raft_consensus.cc:1683] finished
0905 20:00:17.558509 (+ 33us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:17.558643 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35046,"prepare.queue_time_us":174007,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":40630,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":8819,"thread_start_us":148877,"threads_started":2,"wal-append.queue_time_us":153875}
I20250905 20:00:17.574174 28715 leader_election.cc:304] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:17.550309 30675 raft_consensus.cc:2802] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 20:00:17.506569 31079 raft_consensus.cc:3058] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:17.593560 30956 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:17.607031 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 377306us
I20250905 20:00:17.566823 31023 raft_consensus.cc:491] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:17.542922 30660 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:17.500222 30923 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:17.550459 30597 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:17.578419 30741 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 10 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:17.645677 28151 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:36164
W20250905 20:00:17.634862 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:17.571385 31090 raft_consensus.cc:491] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:17.641845 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:17.581387 31059 leader_election.cc:290] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:17.662803 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:17.644327 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:17.664659 28077 raft_consensus.cc:2391] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 6: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:17.584136 31067 consensus_queue.cc:1046] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.055s
I20250905 20:00:17.666827 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:17.668848 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:17.644918 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:17.645666 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:17.670126 28076 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 20:00:17.603533 30675 raft_consensus.cc:491] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:17.676796 28080 raft_consensus.cc:1238] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20250905 20:00:17.632699 31023 raft_consensus.cc:3058] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:17.606369 30985 raft_consensus.cc:491] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:17.662667 28122 leader_election.cc:304] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:17.605173 30850 raft_consensus.cc:513] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:17.643512 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:17.641710 31035 raft_consensus.cc:2747] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250905 20:00:17.666769 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:17.713426 28081 raft_consensus.cc:2391] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 6: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:17.553440 30931 consensus_queue.cc:1035] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.085s
I20250905 20:00:17.714249 28076 raft_consensus.cc:1238] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 6. Current term is 7. Ops: [4.3-6.4]
I20250905 20:00:17.714579 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:17.715435 28084 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 9.
I20250905 20:00:17.678495 28080 raft_consensus.cc:1273] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20250905 20:00:17.688138 31029 consensus_meta.cc:287] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.781s user 0.007s sys 0.004s
I20250905 20:00:17.691314 30923 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:17.692665 30956 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 10 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:17.630884 31074 log.cc:927] Time spent T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.075s user 0.002s sys 0.000s
I20250905 20:00:17.702585 28122 leader_election.cc:304] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:17.644585 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:17.605341 30946 raft_consensus.cc:513] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:17.706025 31057 consensus_meta.cc:287] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.501s user 0.007s sys 0.002s
I20250905 20:00:17.667284 30975 consensus_queue.cc:237] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:17.707967 31011 consensus_meta.cc:287] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.756s user 0.012s sys 0.000s
I20250905 20:00:17.708407 30926 raft_consensus.cc:2747] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:17.708492 31091 raft_consensus.cc:491] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:17.601938 30684 raft_consensus.cc:513] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:17.718267 28157 raft_consensus.cc:1238] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:17.666217 30741 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 10 FOLLOWER]: Advancing to term 11
I20250905 20:00:17.719471 28082 raft_consensus.cc:2391] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 7: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:17.703693 30985 raft_consensus.cc:513] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:17.722505 28122 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:17.583966 31031 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:17.697032 30675 raft_consensus.cc:3058] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
W20250905 20:00:17.706048 30612 consensus_meta.cc:287] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.628s user 0.001s sys 0.009s
I20250905 20:00:17.700902 30537 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:17.701661 31090 raft_consensus.cc:513] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:17.743916 30923 raft_consensus.cc:3058] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:17.836290 30597 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:17.827826 28083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1647) took 1691 ms. Trace:
I20250905 20:00:17.838989 28083 rpcz_store.cc:276] 0905 20:00:16.136485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:16.142953 (+ 6468us) service_pool.cc:225] Handling call
0905 20:00:17.827528 (+1684575us) spinlock_profiling.cc:243] Waited 253 ms on lock 0x7b540023c654. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:17.827732 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":581632384}
I20250905 20:00:17.756296 28159 raft_consensus.cc:1238] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20250905 20:00:17.710388 30969 raft_consensus.cc:3053] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 3 LEADER]: Stepping down as leader of term 3
W20250905 20:00:17.842847 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 120495us
W20250905 20:00:17.808532 31096 log.cc:927] Time spent T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.256s user 0.000s sys 0.001s
I20250905 20:00:17.859829 28078 raft_consensus.cc:2408] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20250905 20:00:17.813148 31000 consensus_meta.cc:287] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.961s user 0.014s sys 0.000s
W20250905 20:00:17.746737 31097 log.cc:927] Time spent T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.143s user 0.000s sys 0.001s
W20250905 20:00:17.745146 30937 consensus_meta.cc:287] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.887s user 0.010s sys 0.003s
I20250905 20:00:17.863876 28122 leader_election.cc:304] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:17.752907 31092 leader_election.cc:290] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 20:00:17.858723 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:17.912518 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 48732us
I20250905 20:00:17.848688 31031 raft_consensus.cc:3058] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Advancing to term 5
W20250905 20:00:17.882249 28124 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.768s user 0.000s sys 0.015s
W20250905 20:00:17.917766 28124 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.803s user 0.000s sys 0.015s
I20250905 20:00:17.913570 28122 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 20:00:17.883867 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 309712us
I20250905 20:00:17.848872 30969 raft_consensus.cc:738] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
W20250905 20:00:17.858866 28051 leader_election.cc:336] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 2.948s (SENT)
I20250905 20:00:17.924070 28051 leader_election.cc:304] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:17.902599 30660 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:17.723615 31029 raft_consensus.cc:513] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:17.848835 31011 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:17.810618 30756 consensus_queue.cc:260] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:17.860037 31103 raft_consensus.cc:2802] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250905 20:00:17.920014 31095 log.cc:927] Time spent T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.147s user 0.001s sys 0.000s
W20250905 20:00:17.911441 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:17.917400 30537 raft_consensus.cc:2747] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:17.938352 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:17.877053 30926 consensus_queue.cc:1046] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20250905 20:00:17.926591 31057 raft_consensus.cc:2802] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250905 20:00:17.928390 31092 raft_consensus.cc:491] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
W20250905 20:00:17.937451 30974 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.001s sys 0.022s
I20250905 20:00:17.930660 30850 leader_election.cc:290] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:17.903249 31000 raft_consensus.cc:513] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:17.911387 30937 raft_consensus.cc:513] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:17.984481 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:17.953900 31043 raft_consensus.cc:491] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:17.928149 30921 consensus_queue.cc:1035] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.026s
W20250905 20:00:18.019016 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:17.988471 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 129612us
I20250905 20:00:18.022547 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:18.015447 31057 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:17.947878 31103 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:17.987131 30756 raft_consensus.cc:3058] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:17.947103 31101 raft_consensus.cc:2747] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250905 20:00:17.947628 30932 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.942s user 0.002s sys 0.011s
I20250905 20:00:18.054731 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:18.055056 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:18.056385 28081 raft_consensus.cc:1273] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250905 20:00:18.059960 28004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 2114) took 2527 ms. Trace:
I20250905 20:00:18.164016 28004 rpcz_store.cc:276] 0905 20:00:15.532776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:16.932176 (+1399400us) service_pool.cc:225] Handling call
0905 20:00:16.932573 (+ 397us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:17.022862 (+ 90289us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:18.059651 (+1036789us) spinlock_profiling.cc:243] Waited 45.3 ms on lock 0x7b54001b0dd4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:18.059884 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":104270592}
W20250905 20:00:18.021700 30902 consensus_meta.cc:287] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.608s user 0.012s sys 0.000s
I20250905 20:00:18.019528 30946 leader_election.cc:290] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 20:00:18.024183 31061 consensus_meta.cc:287] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.637s user 0.010s sys 0.002s
I20250905 20:00:18.070858 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:18.075181 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:17.842471 31091 raft_consensus.cc:513] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:18.183060 28150 raft_consensus.cc:2391] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 8: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:18.139467 28085 raft_consensus.cc:1238] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20250905 20:00:18.027500 31087 raft_consensus.cc:3053] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 LEADER]: Stepping down as leader of term 8
W20250905 20:00:18.026659 30974 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.611s user 0.002s sys 0.022s
I20250905 20:00:18.118775 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:18.073364 31057 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Advancing to term 10
W20250905 20:00:18.099344 30932 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.098s user 0.002s sys 0.011s
I20250905 20:00:18.146420 28076 raft_consensus.cc:3058] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:18.145450 28083 raft_consensus.cc:3058] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:18.025274 31092 raft_consensus.cc:513] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:18.075476 31103 raft_consensus.cc:3058] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:18.184019 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:18.146997 30684 leader_election.cc:290] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:18.069811 31043 raft_consensus.cc:513] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:18.193099 31105 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:18.193473 31108 log.cc:927] Time spent T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.170s user 0.000s sys 0.000s
I20250905 20:00:18.197269 31014 leader_election.cc:290] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:18.165917 31025 consensus_meta.cc:287] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.932s user 0.014s sys 0.000s
I20250905 20:00:18.261734 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:18.263078 28125 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.543s user 0.022s sys 0.057s
W20250905 20:00:18.200618 30663 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.627s user 0.027s sys 0.030s
I20250905 20:00:18.185174 30985 leader_election.cc:290] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:18.288326 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:18.220018 31023 consensus_meta.cc:287] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.505s user 0.011s sys 0.000s
W20250905 20:00:18.307868 28125 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.588s user 0.022s sys 0.057s
I20250905 20:00:18.191267 31061 raft_consensus.cc:513] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:18.205271 31087 raft_consensus.cc:738] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:18.377297 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 20:00:18.391844 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 674053us
I20250905 20:00:18.335074 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:18.181869 30902 raft_consensus.cc:513] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:18.378871 28082 raft_consensus.cc:1238] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:18.262794 30668 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:18.392269 28745 raft_consensus.cc:2466] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 7.
W20250905 20:00:18.392866 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:18.483719 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:18.435581 28076 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 7.
W20250905 20:00:18.484735 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:18.510890 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 112162us
W20250905 20:00:18.381656 31107 log.cc:927] Time spent T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.312s user 0.000s sys 0.001s
I20250905 20:00:18.392280 28150 raft_consensus.cc:2391] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 4: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:18.263594 30660 raft_consensus.cc:491] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:18.380854 31116 log.cc:927] Time spent T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.189s user 0.000s sys 0.000s
I20250905 20:00:18.513144 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:18.377626 30937 leader_election.cc:290] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:18.378152 31118 log.cc:927] Time spent T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.111s user 0.000s sys 0.000s
I20250905 20:00:18.482084 30946 raft_consensus.cc:491] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:18.483667 30977 consensus_meta.cc:287] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 1.014s user 0.011s sys 0.002s
I20250905 20:00:18.355764 31090 leader_election.cc:290] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:18.327543 31025 raft_consensus.cc:2747] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250905 20:00:18.485119 31110 log.cc:927] Time spent T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.337s user 0.001s sys 0.001s
W20250905 20:00:18.381456 30869 consensus_meta.cc:287] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.038s user 0.010s sys 0.000s
I20250905 20:00:18.381007 31043 leader_election.cc:290] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:18.546803 28081 raft_consensus.cc:2466] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
W20250905 20:00:18.328454 30663 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.795s user 0.027s sys 0.030s
I20250905 20:00:18.380652 31011 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 10 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:18.378082 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 20:00:18.513321 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:18.554769 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:18.555714 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:18.485183 31101 raft_consensus.cc:491] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:18.513033 31029 leader_election.cc:290] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:18.507886 30668 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:18.350585 31105 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:18.482132 31023 raft_consensus.cc:513] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:18.780687 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 268827us
I20250905 20:00:18.677137 28083 raft_consensus.cc:2466] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 20:00:18.853691 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:18.596642 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 86828.9us
W20250905 20:00:18.837853 31112 log.cc:927] Time spent T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.482s user 0.001s sys 0.000s
I20250905 20:00:18.846640 30869 raft_consensus.cc:513] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:18.860347 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:18.861294 28077 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1615) took 1213 ms. Trace:
I20250905 20:00:18.861632 28077 rpcz_store.cc:276] 0905 20:00:17.648451 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:17.668588 (+ 20137us) service_pool.cc:225] Handling call
0905 20:00:18.860148 (+1191560us) spinlock_profiling.cc:243] Waited 2.4 ms on lock 0x7b5800204f98. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b7db60 00007fe4a2b7d444 00007fe49c2050c4 00007fe49c205052
0905 20:00:18.861226 (+ 1078us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5520768}
W20250905 20:00:18.867079 28718 connection.cc:232] RPC call timeout handler was delayed by -1.7592664710000463s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17930000000000001s for a response
W20250905 20:00:18.867566 28718 connection.cc:232] RPC call timeout handler was delayed by -1.7390047960001311s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1776s for a response
W20250905 20:00:18.869139 28718 connection.cc:232] RPC call timeout handler was delayed by -1.4910260810000864s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20350000000000001s for a response
W20250905 20:00:18.838704 31113 log.cc:927] Time spent T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.249s user 0.001s sys 0.000s
W20250905 20:00:18.820730 31079 consensus_meta.cc:287] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 1.210s user 0.014s sys 0.000s
I20250905 20:00:18.839725 31124 raft_consensus.cc:491] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:18.888284 28749 raft_consensus.cc:1238] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 6. Current term is 7. Ops: []
W20250905 20:00:18.913451 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 57961us
I20250905 20:00:18.916548 28715 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:18.825445 31091 leader_election.cc:290] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:18.588814 31000 leader_election.cc:290] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:18.554086 30660 raft_consensus.cc:513] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:18.852046 31101 raft_consensus.cc:513] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:18.959585 28084 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1622) took 1298 ms. Trace:
I20250905 20:00:18.927769 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:18.931351 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:18.883983 31090 raft_consensus.cc:491] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:18.884718 31029 raft_consensus.cc:491] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:18.629702 30946 raft_consensus.cc:513] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:18.867157 31092 leader_election.cc:290] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:19.003502 28749 raft_consensus.cc:2391] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 10: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
W20250905 20:00:19.011296 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:19.001371 28079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 333) took 1517 ms. Trace:
I20250905 20:00:19.012513 31131 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:19.012909 30756 consensus_meta.cc:287] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.935s user 0.010s sys 0.001s
I20250905 20:00:19.013810 28002 raft_consensus.cc:2408] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250905 20:00:18.847167 30977 raft_consensus.cc:513] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:18.982944 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 66695.5us
I20250905 20:00:19.012066 28079 rpcz_store.cc:276] 0905 20:00:17.484279 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:17.485285 (+ 1006us) service_pool.cc:225] Handling call
0905 20:00:17.485722 (+ 437us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:17.485953 (+ 231us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:17.485976 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:17.486505 (+ 529us) log.cc:844] Serialized 26 byte log entry
0905 20:00:17.825509 (+339004us) raft_consensus.cc:1643] Marking committed up to 6
0905 20:00:17.825564 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:17.825670 (+ 106us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:19.000984 (+1175314us) raft_consensus.cc:1683] finished
0905 20:00:19.001019 (+ 35us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:19.001277 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":334463,"threads_started":1,"wal-append.queue_time_us":510966}
W20250905 20:00:19.003494 31031 consensus_meta.cc:287] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 1.077s user 0.011s sys 0.000s
I20250905 20:00:18.867798 31011 consensus_queue.cc:1035] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.139s
I20250905 20:00:18.969949 30902 leader_election.cc:290] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:18.992460 30893 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:18.868952 31114 consensus_queue.cc:1046] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.156s
I20250905 20:00:19.013150 31131 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:18.927413 31124 raft_consensus.cc:513] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:19.087190 30668 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:18.997229 28084 rpcz_store.cc:276] 0905 20:00:17.661561 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:17.716301 (+ 54740us) service_pool.cc:225] Handling call
0905 20:00:17.849170 (+132869us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:17.849461 (+ 291us) raft_consensus.cc:1532] Early marking committed up to index 4
0905 20:00:18.208042 (+358581us) log.cc:844] Serialized 14 byte log entry
0905 20:00:18.959025 (+750983us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:18.959119 (+ 94us) raft_consensus.cc:1643] Marking committed up to 4
0905 20:00:18.959205 (+ 86us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:18.959300 (+ 95us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:18.959498 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38817,"prepare.queue_time_us":702549,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":281145,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":28278,"thread_start_us":1054177,"threads_started":3,"wal-append.queue_time_us":775133}
I20250905 20:00:19.252373 30893 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:19.044068 30741 consensus_meta.cc:287] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 1.192s user 0.009s sys 0.005s
I20250905 20:00:19.254065 30741 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:19.254663 28008 raft_consensus.cc:1273] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250905 20:00:19.252494 31031 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:18.689116 31087 consensus_queue.cc:260] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:18.928186 31079 raft_consensus.cc:2747] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:19.002537 30597 raft_consensus.cc:2802] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader election won for term 7
W20250905 20:00:19.247447 28007 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 2096) took 3721 ms (client timeout 2005 ms). Trace:
W20250905 20:00:19.121125 28122 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.027s sys 0.066s
I20250905 20:00:18.982035 31014 raft_consensus.cc:491] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:19.274674 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 118211us
I20250905 20:00:18.998656 31128 raft_consensus.cc:491] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:19.044631 28003 raft_consensus.cc:1238] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 6. Current term is 7. Ops: [6.7-6.7]
W20250905 20:00:19.006865 31117 log.cc:927] Time spent T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.359s user 0.001s sys 0.001s
W20250905 20:00:19.309464 28049 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.565s user 0.008s sys 0.000s
W20250905 20:00:19.379854 28049 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.636s user 0.008s sys 0.002s
W20250905 20:00:19.308137 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 275087us
I20250905 20:00:19.171895 31023 leader_election.cc:290] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 20:00:19.175184 30839 consensus_meta.cc:287] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 1.767s user 0.009s sys 0.006s
I20250905 20:00:19.382097 31014 raft_consensus.cc:513] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:19.289444 30675 consensus_meta.cc:287] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 1.432s user 0.013s sys 0.001s
W20250905 20:00:19.347046 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:19.050678 31127 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:19.271777 31101 leader_election.cc:290] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:19.151247 30932 raft_consensus.cc:491] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:19.002633 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:19.389997 31136 raft_consensus.cc:491] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:19.016391 31029 raft_consensus.cc:513] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
W20250905 20:00:19.348232 30923 consensus_meta.cc:287] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.512s user 0.011s sys 0.003s
W20250905 20:00:19.106117 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 62115.1us
I20250905 20:00:19.349795 30741 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 11 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 20:00:19.218230 31070 log.cc:927] Time spent T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 1.989s user 0.011s sys 0.000s
I20250905 20:00:19.411710 31101 raft_consensus.cc:491] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:19.052102 30927 leader_election.cc:290] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:19.355350 28150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2246) took 1416 ms. Trace:
I20250905 20:00:19.412523 31101 raft_consensus.cc:513] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:19.413189 28150 rpcz_store.cc:276] 0905 20:00:17.938695 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:18.914366 (+975671us) service_pool.cc:225] Handling call
0905 20:00:18.914752 (+ 386us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:18.914984 (+ 232us) raft_consensus.cc:1532] Early marking committed up to index 5
0905 20:00:19.117652 (+202668us) log.cc:844] Serialized 14 byte log entry
0905 20:00:19.354764 (+237112us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:19.354864 (+ 100us) raft_consensus.cc:1643] Marking committed up to 5
0905 20:00:19.354941 (+ 77us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:19.355047 (+ 106us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:19.355238 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15487,"prepare.queue_time_us":459340,"raft.queue_time_us":18866,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":18021,"spinlock_wait_cycles":8781184,"thread_start_us":378465,"threads_started":2}
I20250905 20:00:19.415628 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 20:00:19.358189 28122 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.770s user 0.035s sys 0.071s
I20250905 20:00:19.418684 28155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39586 (request call id 377) took 1739 ms. Trace:
I20250905 20:00:19.419296 28155 rpcz_store.cc:276] 0905 20:00:17.679107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:17.679225 (+ 118us) service_pool.cc:225] Handling call
0905 20:00:17.679532 (+ 307us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:17.680071 (+ 539us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:17.680101 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:17.716875 (+ 36774us) log.cc:844] Serialized 26 byte log entry
0905 20:00:18.054324 (+337449us) raft_consensus.cc:1643] Marking committed up to 5
0905 20:00:18.055215 (+ 891us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:18.055311 (+ 96us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:19.418329 (+1363018us) raft_consensus.cc:1683] finished
0905 20:00:19.418362 (+ 33us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:19.418598 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":716,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":99,"thread_start_us":276727,"threads_started":1,"wal-append.queue_time_us":642268}
W20250905 20:00:19.357332 28007 rpcz_store.cc:269] 0905 20:00:15.525592 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:16.218561 (+692969us) service_pool.cc:225] Handling call
0905 20:00:19.247142 (+3028581us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b5400055fd4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:19.247339 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2787646208}
W20250905 20:00:19.120534 31126 log.cc:927] Time spent T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.084s user 0.000s sys 0.001s
W20250905 20:00:19.058662 30938 consensus_meta.cc:287] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 1.740s user 0.014s sys 0.000s
I20250905 20:00:19.172331 31129 raft_consensus.cc:491] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:19.148397 30240 raft_consensus.cc:3058] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
W20250905 20:00:19.095893 31020 consensus_meta.cc:287] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 1.773s user 0.012s sys 0.002s
I20250905 20:00:19.096005 31025 raft_consensus.cc:491] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:19.227428 31130 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:19.162242 30969 consensus_queue.cc:260] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:19.137280 30977 leader_election.cc:290] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:19.363308 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 20:00:19.290309 30975 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.777s user 0.018s sys 0.056s
I20250905 20:00:19.224774 31033 raft_consensus.cc:3053] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 LEADER]: Stepping down as leader of term 6
I20250905 20:00:19.376737 31128 raft_consensus.cc:513] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:19.011664 31090 raft_consensus.cc:513] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:19.166299 31083 raft_consensus.cc:3053] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 LEADER]: Stepping down as leader of term 7
I20250905 20:00:19.137554 31035 raft_consensus.cc:491] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:19.038941 31105 leader_election.cc:290] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:19.058787 31043 raft_consensus.cc:491] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:19.098815 30974 raft_consensus.cc:491] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:19.118398 31092 consensus_queue.cc:1046] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.261s
I20250905 20:00:19.390022 30932 raft_consensus.cc:513] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:19.409932 31006 raft_consensus.cc:491] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:19.407584 30777 raft_consensus.cc:491] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:19.410509 31047 log.cc:927] Time spent T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 2.735s user 0.000s sys 0.004s
W20250905 20:00:19.427874 31137 log.cc:927] Time spent T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.160s user 0.001s sys 0.000s
I20250905 20:00:19.428041 30927 raft_consensus.cc:491] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:19.382906 30839 raft_consensus.cc:513] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:19.437381 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:19.390720 31136 raft_consensus.cc:513] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:19.019621 31061 leader_election.cc:290] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:19.428704 31127 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:19.446286 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:19.447178 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 20:00:19.467366 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:19.492074 28084 raft_consensus.cc:1273] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250905 20:00:19.784621 28150 raft_consensus.cc:3058] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Advancing to term 10
I20250905 20:00:19.770838 28005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1546) took 4402 ms. Trace:
I20250905 20:00:19.408852 30869 leader_election.cc:290] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:19.815510 30840 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:19.815191 28751 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40904 (request call id 732) took 3392 ms (client timeout 1899 ms). Trace:
I20250905 20:00:19.416424 30923 raft_consensus.cc:2747] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250905 20:00:19.490715 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:19.807325 28078 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1612) took 2163 ms (client timeout 1590 ms). Trace:
W20250905 20:00:19.807358 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 2.51289e+06us
I20250905 20:00:19.785511 28049 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:19.812448 28156 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2227) took 4192 ms. Trace:
I20250905 20:00:19.814030 31010 raft_consensus.cc:491] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:19.814265 31007 raft_consensus.cc:491] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:19.815029 28080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1627) took 1675 ms. Trace:
I20250905 20:00:19.815248 31029 leader_election.cc:290] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:19.408020 31079 raft_consensus.cc:491] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:19.534559 28154 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1686) took 3165 ms. Trace:
I20250905 20:00:18.952358 31121 consensus_queue.cc:1046] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.919s
I20250905 20:00:19.392323 30675 raft_consensus.cc:513] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:19.807518 30927 raft_consensus.cc:513] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:19.478778 30938 raft_consensus.cc:513] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:19.807149 31035 raft_consensus.cc:513] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:19.783279 31103 consensus_meta.cc:287] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.517s user 0.015s sys 0.000s
I20250905 20:00:20.456913 31031 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:20.451076 31090 leader_election.cc:290] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:20.457038 31010 raft_consensus.cc:513] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:19.524829 31149 raft_consensus.cc:491] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:19.807479 30777 raft_consensus.cc:513] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:20.458339 28077 raft_consensus.cc:3058] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:19.808513 31130 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:19.661989 31053 log.cc:927] Time spent T 2e578d82f2b643a795b79abae746575e P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 2.609s user 0.006s sys 0.000s
I20250905 20:00:20.459199 28084 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 337) took 1014 ms. Trace:
I20250905 20:00:19.448239 31134 raft_consensus.cc:491] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:20.137306 28159 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1685) took 3782 ms. Trace:
I20250905 20:00:19.532105 30883 raft_consensus.cc:491] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:20.073324 28715 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.698s user 0.002s sys 0.014s
W20250905 20:00:20.521252 28715 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.159s user 0.008s sys 0.016s
W20250905 20:00:20.521641 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 1.15939e+06us
I20250905 20:00:19.786293 30862 raft_consensus.cc:2747] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:19.816807 28151 raft_consensus.cc:604] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Received request to transfer leadership to TS cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:19.808153 31025 raft_consensus.cc:513] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:20.466158 31007 raft_consensus.cc:513] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:19.849915 30969 raft_consensus.cc:3058] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Advancing to term 4
W20250905 20:00:19.768481 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 325025us
I20250905 20:00:20.443536 31004 raft_consensus.cc:491] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:19.584893 31141 raft_consensus.cc:491] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:20.469300 31124 leader_election.cc:290] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:19.784549 31083 raft_consensus.cc:738] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:19.585656 30893 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:20.472003 30675 leader_election.cc:290] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:19.810534 31043 raft_consensus.cc:513] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:19.814085 31101 leader_election.cc:290] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:19.658322 31139 log.cc:927] Time spent T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20250905 20:00:19.448539 31129 raft_consensus.cc:513] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:19.508931 31057 consensus_meta.cc:287] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.294s user 0.014s sys 0.000s
W20250905 20:00:19.449033 31135 log.cc:927] Time spent T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.125s user 0.000s sys 0.000s
I20250905 20:00:20.462253 31103 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:19.661489 30660 leader_election.cc:290] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:19.543510 31122 log.cc:927] Time spent T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.828s user 0.003s sys 0.000s
I20250905 20:00:19.986318 30971 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
W20250905 20:00:19.786206 30975 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.302s user 0.028s sys 0.072s
I20250905 20:00:19.783154 31033 raft_consensus.cc:738] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:19.985878 30875 raft_consensus.cc:3058] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:19.938935 30950 raft_consensus.cc:491] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:19.823712 31105 raft_consensus.cc:3053] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 LEADER]: Stepping down as leader of term 6
I20250905 20:00:20.461460 30927 leader_election.cc:290] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:20.105710 31014 leader_election.cc:290] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:19.660966 31131 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:20.461764 30938 leader_election.cc:290] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:19.842389 30978 raft_consensus.cc:695] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:19.808022 30974 raft_consensus.cc:513] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:20.500659 31079 raft_consensus.cc:513] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:20.560408 31088 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.002s sys 0.006s
I20250905 20:00:19.933377 31006 raft_consensus.cc:513] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:20.548337 31011 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.138s user 0.016s sys 0.028s
I20250905 20:00:20.534905 31134 raft_consensus.cc:513] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:20.522871 31149 raft_consensus.cc:513] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:20.547838 31128 leader_election.cc:290] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:20.547049 31141 raft_consensus.cc:513] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:20.558161 31057 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:20.556721 31136 leader_election.cc:290] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:20.561703 31127 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:20.564086 31105 raft_consensus.cc:738] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:20.573807 31035 leader_election.cc:290] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 20:00:20.582545 31088 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.648s user 0.003s sys 0.006s
I20250905 20:00:20.572264 31033 consensus_queue.cc:260] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:20.597863 31149 leader_election.cc:290] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:20.565405 31083 consensus_queue.cc:260] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:20.597702 31105 consensus_queue.cc:260] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:20.605085 31141 leader_election.cc:290] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:20.605219 31083 raft_consensus.cc:3058] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:20.605334 31105 raft_consensus.cc:3058] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:20.620162 31057 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 10 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:20.621749 31129 leader_election.cc:290] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:20.604110 31025 leader_election.cc:290] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:20.623097 31043 leader_election.cc:290] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:20.624428 31103 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:20.617023 31033 raft_consensus.cc:3058] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:20.615972 31130 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:20.629554 31134 leader_election.cc:290] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:20.630144 31079 leader_election.cc:290] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:19.888432 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 2.51287e+06us
I20250905 20:00:20.472999 27973 leader_election.cc:304] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 20:00:20.668598 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 195594us
W20250905 20:00:20.677150 27974 connection.cc:232] RPC call timeout handler was delayed by -2.599562639999931s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1899s for a response
W20250905 20:00:19.821005 27975 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:20.700865 27975 connection.cc:232] RPC call timeout handler was delayed by -2.2232838479999373s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1509s for a response
W20250905 20:00:20.701678 27975 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:20.701915 28155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2249) took 2762 ms. Trace:
W20250905 20:00:20.702600 27975 connection.cc:232] RPC call timeout handler was delayed by -1.1369838479999999s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2716s for a response
I20250905 20:00:20.703193 28008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 2117) took 5169 ms. Trace:
I20250905 20:00:20.703727 28008 rpcz_store.cc:276] 0905 20:00:15.533631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:17.003369 (+1469738us) service_pool.cc:225] Handling call
0905 20:00:19.254292 (+2250923us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:19.290188 (+ 35896us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:20.702597 (+1412409us) spinlock_profiling.cc:243] Waited 2.25 s on lock 0x7b5400055fd0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:20.702918 (+ 321us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5175290112}
W20250905 20:00:19.803393 28002 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 2091) took 4279 ms (client timeout 2948 ms). Trace:
W20250905 20:00:20.705811 28002 rpcz_store.cc:269] 0905 20:00:15.524313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:16.431578 (+907265us) service_pool.cc:225] Handling call
0905 20:00:19.544570 (+3112992us) spinlock_profiling.cc:243] Waited 2.39 s on lock 0x7b54001329d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:19.803279 (+258709us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5495459072}
I20250905 20:00:20.462901 28003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 372) took 4219 ms. Trace:
I20250905 20:00:19.808866 28005 rpcz_store.cc:276] 0905 20:00:15.368537 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:16.857727 (+1489190us) service_pool.cc:225] Handling call
0905 20:00:16.858219 (+ 492us) raft_consensus.cc:1469] Updating replica for 3 ops
0905 20:00:16.858678 (+ 459us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:16.858721 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 3 ops
0905 20:00:16.859391 (+ 670us) log.cc:844] Serialized 78 byte log entry
0905 20:00:17.031784 (+172393us) raft_consensus.cc:1643] Marking committed up to 3
0905 20:00:17.095755 (+ 63971us) log.cc:844] Serialized 14 byte log entry
0905 20:00:17.095992 (+ 237us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:17.096071 (+ 79us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:19.213794 (+2117723us) spinlock_profiling.cc:243] Waited 6.03 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49d85891e 00007fe49d857c66 00007fe49d857888 00007fe49d8612c1 00007fe49d861279 00007fe49d861209
0905 20:00:19.412917 (+199123us) raft_consensus.cc:1683] finished
0905 20:00:19.413022 (+ 105us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:19.770740 (+357718us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29328,"prepare.queue_time_us":208764,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":22467,"raft.run_cpu_time_us":402,"raft.run_wall_time_us":36058,"spinlock_wait_cycles":13878528,"thread_start_us":209541,"threads_started":2,"wal-append.queue_time_us":274816,"wal-append.run_cpu_time_us":27638,"wal-append.run_wall_time_us":3301252}
I20250905 20:00:19.843461 28006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 370) took 3650 ms. Trace:
W20250905 20:00:20.469877 28078 rpcz_store.cc:269] 0905 20:00:17.643657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:17.644086 (+ 429us) service_pool.cc:225] Handling call
0905 20:00:19.786295 (+2142209us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b540021fa54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:19.807238 (+ 20943us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":437289472}
I20250905 20:00:20.708057 28003 rpcz_store.cc:276] 0905 20:00:16.243816 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:17.100283 (+856467us) service_pool.cc:225] Handling call
0905 20:00:17.116473 (+ 16190us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:19.377161 (+2260688us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:19.998753 (+621592us) spinlock_profiling.cc:243] Waited 1.93 s on lock 0x7b540027f954. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:20.462810 (+464057us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4433540992}
I20250905 20:00:19.809998 28083 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Advancing to term 11
I20250905 20:00:20.708767 28006 rpcz_store.cc:276] 0905 20:00:16.193376 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:17.064087 (+870711us) service_pool.cc:225] Handling call
0905 20:00:17.064370 (+ 283us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:17.064587 (+ 217us) raft_consensus.cc:1532] Early marking committed up to index 8
0905 20:00:17.165684 (+101097us) log.cc:844] Serialized 14 byte log entry
0905 20:00:19.842903 (+2677219us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:19.843012 (+ 109us) raft_consensus.cc:1643] Marking committed up to 8
0905 20:00:19.843082 (+ 70us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:19.843184 (+ 102us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:19.843380 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2633883,"prepare.queue_time_us":182041,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":6408,"raft.queue_time_us":2583638,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":6504,"thread_start_us":262228,"threads_started":2,"wal-append.queue_time_us":257455,"wal-append.run_cpu_time_us":12796,"wal-append.run_wall_time_us":1489439}
I20250905 20:00:20.710956 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:20.713474 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:20.467674 28080 rpcz_store.cc:276] 0905 20:00:18.140030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:18.140624 (+ 594us) service_pool.cc:225] Handling call
0905 20:00:18.141122 (+ 498us) raft_consensus.cc:1469] Updating replica for 2 ops
0905 20:00:18.311005 (+169883us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:18.311059 (+ 54us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0905 20:00:18.392154 (+ 81095us) log.cc:844] Serialized 52 byte log entry
0905 20:00:18.845005 (+452851us) raft_consensus.cc:1643] Marking committed up to 5
0905 20:00:18.924300 (+ 79295us) log.cc:844] Serialized 14 byte log entry
0905 20:00:18.925059 (+ 759us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:18.925166 (+ 107us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:19.230046 (+304880us) spinlock_profiling.cc:243] Waited 26.5 ms on lock 0x7b180038e780. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49aca43e2 00007fe49aca67b1 00007fe49aca6769 00007fe49aca66f9 00007fe49aca66c1 00007fe49aca59bd 0000000000379254 0000000000379099 00007fe49ac8776a
0905 20:00:19.814682 (+584636us) raft_consensus.cc:1683] finished
0905 20:00:19.814813 (+ 131us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:19.814970 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":60780,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":61,"raft.queue_time_us":288134,"raft.run_cpu_time_us":502,"raft.run_wall_time_us":57741,"spinlock_wait_cycles":60958976,"thread_start_us":450834,"threads_started":2,"wal-append.queue_time_us":436867}
I20250905 20:00:20.515112 28084 rpcz_store.cc:276] 0905 20:00:19.444364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:19.444638 (+ 274us) service_pool.cc:225] Handling call
0905 20:00:19.449983 (+ 5345us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:20.458889 (+1008906us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:20.459124 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:20.137306 28085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1619) took 2487 ms. Trace:
I20250905 20:00:20.716599 28085 rpcz_store.cc:276] 0905 20:00:17.649715 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:17.675886 (+ 26171us) service_pool.cc:225] Handling call
0905 20:00:17.715887 (+ 40001us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:18.183775 (+467888us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:20.136962 (+1953187us) spinlock_profiling.cc:243] Waited 404 ms on lock 0x7b54003aa054. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:20.137224 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":929428736}
W20250905 20:00:20.457352 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 671847us
W20250905 20:00:19.907692 28050 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.163s user 0.032s sys 0.073s
I20250905 20:00:20.717885 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:20.718665 28050 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.974s user 0.035s sys 0.084s
I20250905 20:00:20.721045 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:19.733270 28051 leader_election.cc:336] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 2.005s (SENT)
I20250905 20:00:20.716595 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:20.724133 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 990708us
I20250905 20:00:20.725200 28746 raft_consensus.cc:1238] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 11 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 10. Current term is 11. Ops: []
W20250905 20:00:20.464536 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:20.500283 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:20.727125 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
W20250905 20:00:20.442095 28150 consensus_meta.cc:287] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.633s user 0.006s sys 0.000s
W20250905 20:00:20.727721 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:20.728986 28150 raft_consensus.cc:2408] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
W20250905 20:00:20.729152 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:20.729435 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:20.729758 28150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39586 (request call id 383) took 1615 ms. Trace:
I20250905 20:00:20.526381 28151 raft_consensus.cc:612] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250905 20:00:20.730957 28122 leader_election.cc:304] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:20.731997 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:20.732532 28122 leader_election.cc:304] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:20.733232 28746 raft_consensus.cc:3058] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:20.464967 28156 rpcz_store.cc:276] 0905 20:00:15.619650 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:15.620074 (+ 424us) service_pool.cc:225] Handling call
0905 20:00:15.620828 (+ 754us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:15.640692 (+ 19864us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:15.640738 (+ 46us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:15.974902 (+334164us) log.cc:844] Serialized 26 byte log entry
0905 20:00:16.296142 (+321240us) raft_consensus.cc:1643] Marking committed up to 2
0905 20:00:16.296177 (+ 35us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:16.296248 (+ 71us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:19.382817 (+3086569us) spinlock_profiling.cc:243] Waited 9.43 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49d85891e 00007fe49d857c66 00007fe49d857888 00007fe49d8612c1 00007fe49d861279 00007fe49d861209
0905 20:00:19.812087 (+429270us) raft_consensus.cc:1683] finished
0905 20:00:19.812112 (+ 25us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:19.812378 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19315,"raft.queue_time_us":34195,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":106,"spinlock_wait_cycles":21686656,"thread_start_us":320687,"threads_started":1,"wal-append.queue_time_us":554368}
W20250905 20:00:19.938907 28158 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33752 (request call id 1689) took 3528 ms (client timeout 1509 ms). Trace:
I20250905 20:00:20.560981 30971 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:20.766057 28158 rpcz_store.cc:269] 0905 20:00:16.410257 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:16.410983 (+ 726us) service_pool.cc:225] Handling call
0905 20:00:19.932828 (+3521845us) spinlock_profiling.cc:243] Waited 582 ms on lock 0x7b540036e7d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:19.938782 (+ 5954us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1337983872}
I20250905 20:00:20.767046 31007 leader_election.cc:290] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:20.566632 30950 raft_consensus.cc:513] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:20.768965 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:20.769695 30971 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:20.518004 28159 rpcz_store.cc:276] 0905 20:00:16.355253 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:17.037350 (+682097us) service_pool.cc:225] Handling call
0905 20:00:17.037700 (+ 350us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:17.843584 (+805884us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:20.136962 (+2293378us) spinlock_profiling.cc:243] Waited 718 ms on lock 0x7b54000d1954. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:20.137224 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1651917696}
I20250905 20:00:20.681325 28081 raft_consensus.cc:1273] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250905 20:00:20.770740 31134 raft_consensus.cc:491] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:20.003903 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:20.685637 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:20.475098 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:20.777495 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:20.471673 28154 rpcz_store.cc:276] 0905 20:00:16.368954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:17.049113 (+680159us) service_pool.cc:225] Handling call
0905 20:00:17.049429 (+ 316us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:17.861781 (+812352us) raft_consensus.cc:1532] Early marking committed up to index 4
0905 20:00:17.861832 (+ 51us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:17.861900 (+ 68us) raft_consensus.cc:1643] Marking committed up to 4
0905 20:00:17.861947 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:19.534159 (+1672212us) spinlock_profiling.cc:243] Waited 803 ms on lock 0x7b540026cfd4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:19.534264 (+ 105us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:19.534487 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13190,"raft.queue_time_us":11748,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":37950,"spinlock_wait_cycles":1846425856}
I20250905 20:00:19.784526 28010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 2116) took 4251 ms. Trace:
W20250905 20:00:19.915315 28718 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.870s user 0.056s sys 0.071s
I20250905 20:00:20.791028 30929 raft_consensus.cc:2802] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election won for term 5
W20250905 20:00:20.791481 28718 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.746s user 0.064s sys 0.094s
W20250905 20:00:20.792271 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 1.74747e+06us
I20250905 20:00:20.792054 30978 consensus_queue.cc:237] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:20.695353 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:20.697309 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:20.698971 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:20.715102 28155 rpcz_store.cc:276] 0905 20:00:17.939864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:19.428852 (+1488988us) service_pool.cc:225] Handling call
0905 20:00:19.429243 (+ 391us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:20.687341 (+1258098us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:20.687463 (+ 122us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:20.687578 (+ 115us) raft_consensus.cc:1643] Marking committed up to 5
0905 20:00:20.687662 (+ 84us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:20.701103 (+ 13441us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b5400177354. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:20.701169 (+ 66us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:20.701643 (+ 474us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2890764032}
I20250905 20:00:20.733515 28151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:36164 (request call id 23) took 3087 ms. Trace:
I20250905 20:00:20.796527 28744 raft_consensus.cc:1238] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:20.734448 28153 raft_consensus.cc:2466] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
W20250905 20:00:20.680986 28152 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.343s user 0.008s sys 0.026s
I20250905 20:00:20.734789 28750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40918 (request call id 622) took 2450 ms. Trace:
W20250905 20:00:19.816483 28751 rpcz_store.cc:269] 0905 20:00:16.423229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:16.423394 (+ 165us) service_pool.cc:225] Handling call
0905 20:00:16.841127 (+417733us) spinlock_profiling.cc:243] Waited 1.62 ms on lock 0x7b5800304598. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b7db60 00007fe4a2b7d444 00007fe49c2050c4 00007fe49c205052
0905 20:00:19.814992 (+2973865us) spinlock_profiling.cc:243] Waited 379 ms on lock 0x7b540048c6d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:19.815094 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":876363904}
I20250905 20:00:20.736327 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:20.737583 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
W20250905 20:00:20.727588 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:20.730314 28150 rpcz_store.cc:276] 0905 20:00:19.114523 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:19.437061 (+322538us) service_pool.cc:225] Handling call
0905 20:00:20.729550 (+1292489us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:20.734467 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 20:00:20.516611 30597 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.124s user 0.009s sys 0.023s
I20250905 20:00:20.498569 30773 raft_consensus.cc:2802] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250905 20:00:20.801074 30597 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.412s user 0.009s sys 0.024s
I20250905 20:00:20.801512 30773 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
W20250905 20:00:20.801893 28152 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.508s user 0.008s sys 0.026s
I20250905 20:00:20.802235 30773 raft_consensus.cc:3058] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:19.816130 30840 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:20.803010 28010 rpcz_store.cc:276] 0905 20:00:15.533345 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:16.990059 (+1456714us) service_pool.cc:225] Handling call
0905 20:00:16.990386 (+ 327us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:16.993512 (+ 3126us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:16.993543 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:16.994015 (+ 472us) log.cc:844] Serialized 26 byte log entry
0905 20:00:17.094057 (+100042us) raft_consensus.cc:1643] Marking committed up to 5
0905 20:00:17.269387 (+175330us) log.cc:844] Serialized 14 byte log entry
0905 20:00:19.545000 (+2275613us) spinlock_profiling.cc:243] Waited 2.81 ms on lock 0x7b18000314e0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49ac19f4a 00007fe4a4107409 00007fe49aadb3e0 00007fe49aca14e4 00007fe49aca1455 00007fe49dc11020 00007fe49dc236c6 00007fe49dc34237 00007fe49dc341b8
0905 20:00:19.784053 (+239053us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:19.784198 (+ 145us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:19.784239 (+ 41us) raft_consensus.cc:1683] finished
0905 20:00:19.784316 (+ 77us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:19.784469 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":357417,"prepare.queue_time_us":155453,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":2542359,"raft.run_cpu_time_us":297,"raft.run_wall_time_us":18007,"spinlock_wait_cycles":6464256,"thread_start_us":151103,"threads_started":2,"wal-append.queue_time_us":165133,"wal-append.run_cpu_time_us":11326,"wal-append.run_wall_time_us":1588445}
I20250905 20:00:20.805855 30840 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:20.525545 30883 raft_consensus.cc:513] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:20.145148 30839 leader_election.cc:290] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:20.477382 30932 leader_election.cc:290] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:20.807288 31010 leader_election.cc:290] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:20.809144 30883 leader_election.cc:290] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:20.810518 31006 leader_election.cc:290] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:20.801271 30835 raft_consensus.cc:491] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:20.809890 28157 raft_consensus.cc:2466] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 6.
I20250905 20:00:20.746603 30537 raft_consensus.cc:491] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:20.747627 30877 raft_consensus.cc:491] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
W20250905 20:00:20.749861 28050 connection.cc:232] RPC call timeout handler was delayed by -1.7646642260000363s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1876s for a response
I20250905 20:00:20.747999 31031 raft_consensus.cc:3053] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 LEADER]: Stepping down as leader of term 5
W20250905 20:00:20.583040 31011 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.173s user 0.016s sys 0.028s
I20250905 20:00:20.770041 28158 raft_consensus.cc:3058] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:20.756701 30974 leader_election.cc:290] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:20.778604 31130 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:20.779219 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:20.777527 28749 raft_consensus.cc:2391] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 7: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 20:00:20.779773 30775 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:20.774186 31134 raft_consensus.cc:513] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:20.780961 28005 raft_consensus.cc:2466] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
I20250905 20:00:20.781225 30849 raft_consensus.cc:491] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:20.781679 30950 leader_election.cc:290] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:20.782126 30850 raft_consensus.cc:491] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:20.829037 30775 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:20.829824 30850 raft_consensus.cc:513] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:20.830194 30849 raft_consensus.cc:513] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:20.831696 30775 leader_election.cc:290] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:20.832077 30850 leader_election.cc:290] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:20.832705 30849 leader_election.cc:290] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:20.783735 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:20.819952 31031 raft_consensus.cc:738] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:20.833201 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:20.833678 28159 raft_consensus.cc:3058] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 7
W20250905 20:00:20.766084 31157 log.cc:927] Time spent T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.132s user 0.000s sys 0.001s
I20250905 20:00:20.834654 31031 consensus_queue.cc:260] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:20.836089 31134 leader_election.cc:290] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:20.836663 31031 raft_consensus.cc:3058] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:20.789043 30884 raft_consensus.cc:2802] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:20.798249 28007 raft_consensus.cc:3058] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:20.838771 30884 raft_consensus.cc:491] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:20.839406 30884 raft_consensus.cc:3058] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:20.839013 30877 raft_consensus.cc:513] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:20.801565 30597 raft_consensus.cc:695] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:20.841027 30537 raft_consensus.cc:513] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:20.842576 30597 consensus_queue.cc:237] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:20.745805 30148 raft_consensus.cc:2802] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:20.843730 30148 raft_consensus.cc:491] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:20.844194 30148 raft_consensus.cc:3058] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:20.540297 31004 raft_consensus.cc:513] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:20.847878 30537 leader_election.cc:290] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:20.809525 28150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2250) took 2868 ms. Trace:
I20250905 20:00:20.854558 28083 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 11.
W20250905 20:00:20.855543 28083 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50660 (request call id 340) took 1409 ms (client timeout 1736 ms). Trace:
I20250905 20:00:20.855849 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:20.855989 28083 rpcz_store.cc:269] 0905 20:00:19.445881 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:19.446024 (+ 143us) service_pool.cc:225] Handling call
0905 20:00:20.855354 (+1409330us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 20:00:20.823835 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 63415.3us
I20250905 20:00:20.797338 28151 rpcz_store.cc:276] 0905 20:00:17.645392 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:17.645549 (+ 157us) service_pool.cc:225] Handling call
0905 20:00:20.733066 (+3087517us) spinlock_profiling.cc:243] Waited 2.15 s on lock 0x7b5400317c54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc13813 00007fe4a2b801ca 00007fe49c2131c4 00007fe49c213152 00007fe49c213081 00007fe49c212ffc 00007fe49c2122b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:20.733239 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4955982080}
I20250905 20:00:20.813124 30835 raft_consensus.cc:513] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:20.859012 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:20.859839 28151 raft_consensus.cc:3058] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:20.860689 30888 raft_consensus.cc:3053] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 6 LEADER]: Stepping down as leader of term 6
I20250905 20:00:20.861179 30888 raft_consensus.cc:738] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:20.797989 30929 raft_consensus.cc:695] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:20.829231 30956 consensus_queue.cc:1046] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20250905 20:00:20.782563 30684 raft_consensus.cc:491] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:20.784175 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:20.834362 28156 raft_consensus.cc:2391] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 6: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:20.874676 28156 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33752 (request call id 1699) took 1052 ms. Trace:
I20250905 20:00:20.875123 28156 rpcz_store.cc:276] 0905 20:00:19.822036 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:20.790532 (+968496us) service_pool.cc:225] Handling call
0905 20:00:20.874559 (+ 84027us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:20.875465 30877 leader_election.cc:290] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:20.838415 28154 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2251) took 2896 ms. Trace:
I20250905 20:00:20.823184 30777 leader_election.cc:290] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:20.876711 28154 rpcz_store.cc:276] 0905 20:00:17.942086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:20.836989 (+2894903us) service_pool.cc:225] Handling call
0905 20:00:20.837506 (+ 517us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:20.837935 (+ 429us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:20.837960 (+ 25us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:20.838025 (+ 65us) raft_consensus.cc:1643] Marking committed up to 4
0905 20:00:20.838054 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:20.838112 (+ 58us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:20.838243 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:20.878160 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:20.806103 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:20.879601 28002 raft_consensus.cc:2466] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 7.
I20250905 20:00:20.880661 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:20.842762 31130 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:20.881422 28009 raft_consensus.cc:3058] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Advancing to term 7
W20250905 20:00:20.881718 28153 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33752 (request call id 1681) took 4581 ms (client timeout 2716 ms). Trace:
I20250905 20:00:20.798274 28750 rpcz_store.cc:276] 0905 20:00:18.284269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:18.284433 (+ 164us) service_pool.cc:225] Handling call
0905 20:00:18.284825 (+ 392us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:18.394821 (+109996us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:18.394851 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:18.395359 (+ 508us) log.cc:844] Serialized 26 byte log entry
0905 20:00:18.912277 (+516918us) raft_consensus.cc:1643] Marking committed up to 6
0905 20:00:19.109514 (+197237us) log.cc:844] Serialized 14 byte log entry
0905 20:00:19.109953 (+ 439us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:19.110060 (+ 107us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:20.734192 (+1624132us) raft_consensus.cc:1683] finished
0905 20:00:20.734240 (+ 48us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:20.734530 (+ 290us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22265,"prepare.queue_time_us":903853,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":2142,"raft.queue_time_us":199155,"raft.run_cpu_time_us":9820,"raft.run_wall_time_us":43918,"thread_start_us":521690,"threads_started":2,"wal-append.queue_time_us":625681}
I20250905 20:00:20.882982 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:20.854868 28150 rpcz_store.cc:276] 0905 20:00:17.941333 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:20.808100 (+2866767us) service_pool.cc:225] Handling call
0905 20:00:20.808638 (+ 538us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:20.809107 (+ 469us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:20.809141 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:20.809222 (+ 81us) raft_consensus.cc:1643] Marking committed up to 8
0905 20:00:20.809261 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:20.809319 (+ 58us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:20.809444 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:20.848389 31004 leader_election.cc:290] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:20.885308 31130 leader_election.cc:290] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:20.873406 30684 raft_consensus.cc:513] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:20.888499 30684 leader_election.cc:290] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:20.889546 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:20.899398 30929 consensus_queue.cc:237] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:20.900199 28156 raft_consensus.cc:2466] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
I20250905 20:00:20.870584 28158 raft_consensus.cc:2466] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 7.
I20250905 20:00:20.813995 28122 leader_election.cc:304] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
W20250905 20:00:20.902897 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 142387us
W20250905 20:00:20.903148 28125 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.833s (SENT) [suppressed 19 similar messages]
I20250905 20:00:20.905354 31087 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Advancing to term 9
I20250905 20:00:20.878197 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:20.907001 28006 raft_consensus.cc:2466] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 7.
I20250905 20:00:20.910161 30926 raft_consensus.cc:2747] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250905 20:00:20.910847 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:20.884754 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:20.912806 28085 raft_consensus.cc:1238] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:20.913415 28748 raft_consensus.cc:2466] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
I20250905 20:00:20.901053 28158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33752 (request call id 1701) took 1057 ms. Trace:
I20250905 20:00:20.915019 28084 raft_consensus.cc:1238] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 11 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 10. Current term is 11. Ops: []
I20250905 20:00:20.910725 30875 raft_consensus.cc:491] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:20.917183 28081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1630) took 1469 ms. Trace:
I20250905 20:00:20.916975 30875 raft_consensus.cc:513] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:20.917824 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:20.918771 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:20.919356 31151 raft_consensus.cc:491] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:20.919703 28155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33752 (request call id 1698) took 1098 ms. Trace:
I20250905 20:00:20.919958 31151 raft_consensus.cc:513] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:20.921716 30777 raft_consensus.cc:491] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:20.917102 28080 raft_consensus.cc:1238] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
W20250905 20:00:20.922807 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:20.923897 30777 raft_consensus.cc:513] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:20.925011 31114 raft_consensus.cc:491] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:20.925309 30969 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:20.925734 31114 raft_consensus.cc:513] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:20.925998 30969 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:20.889878 30956 raft_consensus.cc:3053] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 10 LEADER]: Stepping down as leader of term 10
I20250905 20:00:20.892098 30849 consensus_queue.cc:1046] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20250905 20:00:20.891302 28749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40918 (request call id 627) took 1531 ms (client timeout 1791 ms). Trace:
I20250905 20:00:20.894645 31093 raft_consensus.cc:3053] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 7 LEADER]: Stepping down as leader of term 7
I20250905 20:00:20.884606 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:20.901384 28154 raft_consensus.cc:2408] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250905 20:00:20.920285 28155 rpcz_store.cc:276] 0905 20:00:19.821691 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:20.799984 (+978293us) service_pool.cc:225] Handling call
0905 20:00:20.919637 (+119653us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:20.919559 28007 raft_consensus.cc:2466] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 7.
I20250905 20:00:20.890851 30888 consensus_queue.cc:260] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:20.893297 27974 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:20.932555 28159 raft_consensus.cc:2466] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 7.
I20250905 20:00:20.939901 28159 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33752 (request call id 1700) took 1117 ms. Trace:
I20250905 20:00:20.915066 28158 rpcz_store.cc:276] 0905 20:00:19.844010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:20.768223 (+924213us) service_pool.cc:225] Handling call
0905 20:00:20.900974 (+132751us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 20:00:20.882258 28153 rpcz_store.cc:269] 0905 20:00:16.300379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:16.385516 (+ 85137us) service_pool.cc:225] Handling call
0905 20:00:20.881470 (+4495954us) spinlock_profiling.cc:243] Waited 4.31 s on lock 0x7b5400317c54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:20.881631 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9910726016}
I20250905 20:00:20.940352 28159 rpcz_store.cc:276] 0905 20:00:19.822338 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:20.771204 (+948866us) service_pool.cc:225] Handling call
0905 20:00:20.939821 (+168617us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 20:00:20.929044 28749 rpcz_store.cc:269] 0905 20:00:19.360523 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:19.362940 (+ 2417us) service_pool.cc:225] Handling call
0905 20:00:20.891016 (+1528076us) spinlock_profiling.cc:243] Waited 965 ms on lock 0x7b540049d4d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:20.891210 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2218986752}
I20250905 20:00:20.917560 28081 rpcz_store.cc:276] 0905 20:00:19.447409 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:19.447628 (+ 219us) service_pool.cc:225] Handling call
0905 20:00:19.530541 (+ 82913us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:20.770951 (+1240410us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:20.915716 (+144765us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b540023ad54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:20.917105 (+ 1389us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2642618112}
I20250905 20:00:20.962218 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:20.962903 28077 raft_consensus.cc:2466] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6.
I20250905 20:00:20.963097 28751 raft_consensus.cc:2466] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 7.
I20250905 20:00:20.982540 28077 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50660 (request call id 339) took 1537 ms. Trace:
I20250905 20:00:20.982957 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:20.983675 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:20.926416 30777 leader_election.cc:290] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:20.928377 31114 leader_election.cc:290] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:20.983098 28077 rpcz_store.cc:276] 0905 20:00:19.445501 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:19.446818 (+ 1317us) service_pool.cc:225] Handling call
0905 20:00:20.982403 (+1535585us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:20.929212 31093 raft_consensus.cc:738] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:20.926400 30835 leader_election.cc:290] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:21.000075 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:20.928750 30956 raft_consensus.cc:738] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:21.027953 28083 raft_consensus.cc:1238] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:20.955000 28009 raft_consensus.cc:2408] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20250905 20:00:20.928341 30969 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:21.038867 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:21.041235 28155 raft_consensus.cc:3058] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:21.030678 28750 raft_consensus.cc:1238] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:21.031677 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:21.044178 28156 raft_consensus.cc:2391] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 6: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:20.984582 30875 leader_election.cc:290] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:21.019987 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:21.032596 28748 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40918 (request call id 621) took 2748 ms (client timeout 3097 ms). Trace:
W20250905 20:00:21.048465 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 155293us
I20250905 20:00:21.048804 28004 raft_consensus.cc:1273] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250905 20:00:21.049000 31093 consensus_queue.cc:260] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:21.050237 28010 raft_consensus.cc:2466] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 7.
I20250905 20:00:21.050411 31093 raft_consensus.cc:3058] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:21.055120 28005 raft_consensus.cc:2466] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 8.
I20250905 20:00:21.057185 28081 raft_consensus.cc:2466] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
I20250905 20:00:21.058676 31084 raft_consensus.cc:491] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:21.058799 27974 leader_election.cc:304] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:21.059227 30956 consensus_queue.cc:260] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:21.059559 31084 raft_consensus.cc:513] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:21.060912 31083 raft_consensus.cc:2802] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 20:00:21.061383 31083 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:21.061758 31083 raft_consensus.cc:3058] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:21.048812 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:21.063354 30684 consensus_queue.cc:1046] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250905 20:00:21.064113 28153 raft_consensus.cc:2408] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20250905 20:00:21.064764 30775 raft_consensus.cc:3053] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Stepping down as leader of term 5
I20250905 20:00:21.065073 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:21.065172 30775 raft_consensus.cc:738] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:21.066025 28153 raft_consensus.cc:2466] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
I20250905 20:00:21.031225 30862 raft_consensus.cc:3053] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Stepping down as leader of term 4
I20250905 20:00:21.066761 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:21.067881 28079 raft_consensus.cc:2466] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 7.
I20250905 20:00:21.032318 30773 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:21.066706 30775 consensus_queue.cc:260] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:21.070415 30773 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:21.047432 28748 rpcz_store.cc:269] 0905 20:00:18.283938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:18.334600 (+ 50662us) service_pool.cc:225] Handling call
0905 20:00:21.032350 (+2697750us) spinlock_profiling.cc:243] Waited 2.52 s on lock 0x7b54004e1954. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:21.032508 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5795557760}
I20250905 20:00:21.071026 30777 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:21.071941 30777 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:21.074280 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:21.077185 30777 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:21.079478 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:21.081022 30775 raft_consensus.cc:3058] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:21.036581 30148 raft_consensus.cc:513] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:21.074848 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:21.076143 31131 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:21.083191 30888 raft_consensus.cc:3058] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:21.083683 31131 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:21.063460 30956 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 10 FOLLOWER]: Advancing to term 11
I20250905 20:00:21.049608 30849 raft_consensus.cc:2802] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 20:00:21.087594 30849 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:21.088080 30849 raft_consensus.cc:3058] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:21.088346 31131 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:21.089493 28154 raft_consensus.cc:1238] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 7. Current term is 8. Ops: []
I20250905 20:00:21.090175 28154 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2253) took 3147 ms. Trace:
I20250905 20:00:21.055246 28080 raft_consensus.cc:1238] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20250905 20:00:21.093029 28081 raft_consensus.cc:3058] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:21.092383 28154 rpcz_store.cc:276] 0905 20:00:17.943190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.088803 (+3145613us) service_pool.cc:225] Handling call
0905 20:00:21.089241 (+ 438us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:21.089937 (+ 696us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:21.090119 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 20:00:21.095866 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:21.096602 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:21.099349 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:21.099759 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:21.100364 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:21.110155 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:21.113075 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:21.113500 28744 raft_consensus.cc:1238] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:21.113960 28008 raft_consensus.cc:2391] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 7: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:21.026666 31099 raft_consensus.cc:491] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:21.035621 31151 leader_election.cc:290] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:21.082129 31031 consensus_queue.cc:1035] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.164s
I20250905 20:00:21.641448 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:22.130692 28153 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s user 0.014s sys 0.011s
I20250905 20:00:21.550415 28082 raft_consensus.cc:1238] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 7. Current term is 8. Ops: []
W20250905 20:00:21.171991 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 268976us
I20250905 20:00:21.281430 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:21.399250 28155 raft_consensus.cc:2466] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 6.
I20250905 20:00:21.115657 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:21.400491 28122 leader_election.cc:336] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.590s (SENT)
W20250905 20:00:21.725250 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:21.332943 28080 raft_consensus.cc:1238] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:21.125837 28150 raft_consensus.cc:1273] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250905 20:00:21.281499 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:21.348990 28076 raft_consensus.cc:1238] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 6. Current term is 7. Ops: [4.5-6.7]
I20250905 20:00:21.281807 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
W20250905 20:00:21.566152 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:21.605085 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:21.546833 28003 raft_consensus.cc:1273] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250905 20:00:21.389488 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:21.735809 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:21.579665 28077 raft_consensus.cc:1273] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250905 20:00:21.176549 28081 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 20:00:21.114257 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:21.272066 28157 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 2245) took 3322 ms (client timeout 1876 ms). Trace:
W20250905 20:00:22.350194 28153 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.801s user 0.020s sys 0.015s
I20250905 20:00:22.353400 28150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39586 (request call id 380) took 3305 ms. Trace:
I20250905 20:00:21.589233 28159 raft_consensus.cc:1273] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250905 20:00:21.114043 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:22.360311 28159 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2254) took 4280 ms. Trace:
I20250905 20:00:21.117116 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:22.354694 28006 raft_consensus.cc:2466] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 7.
I20250905 20:00:22.359570 28150 rpcz_store.cc:276] 0905 20:00:19.047623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.124785 (+2077162us) service_pool.cc:225] Handling call
0905 20:00:21.125321 (+ 536us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:22.353167 (+1227846us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:22.353338 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 20:00:21.667281 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:21.643404 31093 consensus_meta.cc:287] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.592s user 0.014s sys 0.003s
I20250905 20:00:21.092509 30862 raft_consensus.cc:738] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:21.117748 30946 leader_election.cc:290] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:22.364588 28008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 2133) took 1485 ms. Trace:
I20250905 20:00:22.365216 28008 rpcz_store.cc:276] 0905 20:00:20.879207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.045316 (+166109us) service_pool.cc:225] Handling call
0905 20:00:22.364491 (+1319175us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:21.279803 27908 catalog_manager.cc:5582] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 2 to 4, leader changed from 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) to 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132). New cstate: current_term: 4 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 20:00:22.365217 30862 consensus_queue.cc:260] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:22.366900 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:22.367569 30862 raft_consensus.cc:3058] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:22.368333 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:22.368072 27908 catalog_manager.cc:5582] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 20:00:22.368965 28008 raft_consensus.cc:2374] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 7. Re-sending same reply.
I20250905 20:00:22.369943 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:22.352313 28010 raft_consensus.cc:2466] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 7.
I20250905 20:00:21.540916 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:22.370787 28010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 2139) took 1279 ms. Trace:
I20250905 20:00:22.371133 28010 rpcz_store.cc:276] 0905 20:00:21.091167 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.102050 (+ 10883us) service_pool.cc:225] Handling call
0905 20:00:22.370732 (+1268682us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.371497 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:22.371726 30684 raft_consensus.cc:491] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:22.372040 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:22.372176 28004 raft_consensus.cc:2466] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 6.
I20250905 20:00:22.372761 28010 raft_consensus.cc:2466] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
I20250905 20:00:22.372596 30684 raft_consensus.cc:513] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 20:00:22.373651 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:22.374087 28076 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1636) took 1026 ms. Trace:
I20250905 20:00:22.374465 28076 rpcz_store.cc:276] 0905 20:00:21.347203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.347452 (+ 249us) service_pool.cc:225] Handling call
0905 20:00:21.348258 (+ 806us) raft_consensus.cc:1469] Updating replica for 3 ops
0905 20:00:22.373676 (+1025418us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:22.374011 (+ 335us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.375222 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:22.354032 28125 connection.cc:232] RPC call timeout handler was delayed by -1.1658603250000397s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21070000000000003s for a response
W20250905 20:00:22.376398 28125 connection.cc:232] RPC call timeout handler was delayed by -1.1649603250000382s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21080000000000002s for a response
W20250905 20:00:21.681315 28151 consensus_meta.cc:287] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.726s user 0.009s sys 0.003s
I20250905 20:00:22.373171 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:21.155440 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:22.377137 28151 raft_consensus.cc:2466] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
W20250905 20:00:21.123394 28123 leader_election.cc:336] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299): Timed out: RequestConsensusVote RPC to 127.26.58.132:44299 timed out after 1.691s (SENT)
W20250905 20:00:22.357448 28157 rpcz_store.cc:269] 0905 20:00:17.938242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:18.070454 (+132212us) service_pool.cc:225] Handling call
0905 20:00:21.169721 (+3099267us) spinlock_profiling.cc:243] Waited 2.32 s on lock 0x7b5400187754. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:21.260404 (+ 90683us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5326676352}
I20250905 20:00:22.377884 28154 raft_consensus.cc:1238] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 4. Current term is 5. Ops: []
W20250905 20:00:21.145113 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:22.379117 28151 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 2259) took 1533 ms (client timeout 1958 ms). Trace:
W20250905 20:00:22.353013 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:21.116258 28750 raft_consensus.cc:1238] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 11 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 10. Current term is 11. Ops: []
I20250905 20:00:21.129669 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 20:00:21.649545 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 107105us
I20250905 20:00:22.380892 28080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1669) took 1035 ms. Trace:
I20250905 20:00:22.360939 28159 rpcz_store.cc:276] 0905 20:00:18.080292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.119202 (+3038910us) service_pool.cc:225] Handling call
0905 20:00:21.539114 (+419912us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:22.359977 (+820863us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:22.360226 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 20:00:22.381244 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:22.382797 28010 raft_consensus.cc:2408] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20250905 20:00:22.380913 28155 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 2269) took 1484 ms (client timeout 1808 ms). Trace:
W20250905 20:00:22.384245 28155 rpcz_store.cc:269] 0905 20:00:20.896645 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:20.942112 (+ 45467us) service_pool.cc:225] Handling call
0905 20:00:22.380857 (+1438745us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 20:00:22.382933 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:22.379603 28151 rpcz_store.cc:269] 0905 20:00:20.845838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:20.857433 (+ 11595us) service_pool.cc:225] Handling call
0905 20:00:22.379055 (+1521622us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.381727 28077 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1668) took 1040 ms. Trace:
I20250905 20:00:22.386693 28077 rpcz_store.cc:276] 0905 20:00:21.317448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.317648 (+ 200us) service_pool.cc:225] Handling call
0905 20:00:21.578791 (+261143us) spinlock_profiling.cc:243] Waited 63.1 ms on lock 0x7b5800204f98. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b91f60 00007fe4a2b7d1a9 00007fe4a2b7cc52 00007fe49c202b44
0905 20:00:21.579133 (+ 342us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:22.356851 (+777718us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:22.357271 (+ 420us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":145192192}
I20250905 20:00:22.375801 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:21.117884 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:22.364527 28153 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1695) took 4421 ms. Trace:
I20250905 20:00:21.589648 28158 raft_consensus.cc:2466] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 6.
W20250905 20:00:21.621174 27904 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34446 (request call id 239) took 28405 ms (client timeout 15000 ms). Trace:
I20250905 20:00:21.083271 30148 leader_election.cc:290] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:22.390571 28158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 2268) took 1494 ms. Trace:
I20250905 20:00:21.241043 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:21.558398 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:21.313112 28152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1677) took 5022 ms. Trace:
W20250905 20:00:21.775400 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 87812.2us
I20250905 20:00:21.349246 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:21.354269 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:21.951079 31158 log.cc:927] Time spent T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.342s user 0.000s sys 0.002s
I20250905 20:00:22.394443 28122 leader_election.cc:304] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:22.395179 28003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 2143) took 1009 ms. Trace:
I20250905 20:00:22.426712 28005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 2142) took 1158 ms. Trace:
I20250905 20:00:22.437417 28003 rpcz_store.cc:276] 0905 20:00:21.346312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.346502 (+ 190us) service_pool.cc:225] Handling call
0905 20:00:21.545491 (+198989us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:22.355028 (+809537us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:22.355288 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.428824 28009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 2140) took 1337 ms. Trace:
I20250905 20:00:22.393877 28007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 2141) took 1265 ms. Trace:
W20250905 20:00:22.376828 28125 connection.cc:232] RPC call timeout handler was delayed by -1.0488603250000779s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2237s for a response
I20250905 20:00:22.389508 28076 raft_consensus.cc:2466] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
I20250905 20:00:22.439218 28076 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1643) took 1087 ms. Trace:
I20250905 20:00:21.668026 30775 raft_consensus.cc:491] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:22.439625 28076 rpcz_store.cc:276] 0905 20:00:21.351940 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.375514 (+1023574us) service_pool.cc:225] Handling call
0905 20:00:22.439155 (+ 63641us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.380499 28748 raft_consensus.cc:2466] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
I20250905 20:00:22.398723 28123 leader_election.cc:304] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:22.438756 28007 rpcz_store.cc:276] 0905 20:00:21.091870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.092164 (+ 294us) service_pool.cc:225] Handling call
0905 20:00:22.357451 (+1265287us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.440577 28748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40926 (request call id 529) took 1359 ms. Trace:
I20250905 20:00:21.543983 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:22.440989 28748 rpcz_store.cc:276] 0905 20:00:21.081412 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.101694 (+ 20282us) service_pool.cc:225] Handling call
0905 20:00:22.440516 (+1338822us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.398125 28747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40926 (request call id 533) took 1032 ms. Trace:
I20250905 20:00:22.398536 28154 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1697) took 4452 ms. Trace:
I20250905 20:00:22.403445 28158 rpcz_store.cc:276] 0905 20:00:20.896296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:20.955451 (+ 59155us) service_pool.cc:225] Handling call
0905 20:00:22.390507 (+1435056us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.398200 28750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40904 (request call id 752) took 1282 ms. Trace:
I20250905 20:00:22.398586 28749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40918 (request call id 630) took 1428 ms. Trace:
I20250905 20:00:22.443317 28749 rpcz_store.cc:276] 0905 20:00:20.970910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.155027 (+184117us) service_pool.cc:225] Handling call
0905 20:00:22.398516 (+1243489us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.381356 28080 rpcz_store.cc:276] 0905 20:00:21.317888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.318046 (+ 158us) service_pool.cc:225] Handling call
0905 20:00:21.332685 (+ 14639us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:22.353120 (+1020435us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:22.353338 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.418073 28002 raft_consensus.cc:2466] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 6.
I20250905 20:00:22.418645 28078 raft_consensus.cc:2466] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 4.
I20250905 20:00:22.412146 30148 raft_consensus.cc:491] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:22.444458 28002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 2144) took 1069 ms. Trace:
I20250905 20:00:22.441869 28747 rpcz_store.cc:276] 0905 20:00:21.347020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.670930 (+323910us) service_pool.cc:225] Handling call
0905 20:00:22.379021 (+708091us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 20:00:21.480525 28746 consensus_meta.cc:287] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.692s user 0.007s sys 0.003s
I20250905 20:00:22.419102 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:22.346431 31099 raft_consensus.cc:513] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:22.424340 30684 leader_election.cc:290] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:22.425520 28159 raft_consensus.cc:1238] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:22.426476 28157 raft_consensus.cc:1238] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:22.426792 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:22.427109 28745 raft_consensus.cc:2466] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:22.390223 28153 rpcz_store.cc:276] 0905 20:00:17.943537 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.092423 (+3148886us) service_pool.cc:225] Handling call
0905 20:00:21.151648 (+ 59225us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:21.152035 (+ 387us) raft_consensus.cc:1532] Early marking committed up to index 3
0905 20:00:21.271893 (+119858us) log.cc:844] Serialized 14 byte log entry
0905 20:00:22.363973 (+1092080us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:22.364039 (+ 66us) raft_consensus.cc:1643] Marking committed up to 3
0905 20:00:22.364099 (+ 60us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:22.364173 (+ 74us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:22.364461 (+ 288us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7134,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":146,"thread_start_us":927924,"threads_started":2}
I20250905 20:00:22.427631 28083 raft_consensus.cc:3058] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:22.428193 28084 raft_consensus.cc:3058] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:22.427955 31151 consensus_queue.cc:1035] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.030s
I20250905 20:00:22.428575 28744 raft_consensus.cc:3058] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:22.428562 28752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40918 (request call id 629) took 1503 ms. Trace:
W20250905 20:00:21.900424 30956 consensus_meta.cc:287] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.813s user 0.016s sys 0.001s
I20250905 20:00:22.460817 28077 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1640) took 1109 ms. Trace:
I20250905 20:00:22.461385 28077 rpcz_store.cc:276] 0905 20:00:21.351007 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.388252 (+1037245us) service_pool.cc:225] Handling call
0905 20:00:22.460732 (+ 72480us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.462792 28159 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39586 (request call id 382) took 3334 ms. Trace:
I20250905 20:00:22.463307 28159 rpcz_store.cc:276] 0905 20:00:19.114116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.424882 (+3310766us) service_pool.cc:225] Handling call
0905 20:00:22.425305 (+ 423us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:22.447960 (+ 22655us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:22.448195 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.429273 28155 raft_consensus.cc:1273] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250905 20:00:22.403633 28152 rpcz_store.cc:276] 0905 20:00:16.291041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:17.017359 (+726318us) service_pool.cc:225] Handling call
0905 20:00:17.017796 (+ 437us) raft_consensus.cc:1469] Updating replica for 2 ops
0905 20:00:17.018242 (+ 446us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:17.018284 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0905 20:00:17.048114 (+ 29830us) log.cc:844] Serialized 52 byte log entry
0905 20:00:17.353580 (+305466us) raft_consensus.cc:1643] Marking committed up to 5
0905 20:00:17.408300 (+ 54720us) log.cc:844] Serialized 14 byte log entry
0905 20:00:19.249993 (+1841693us) spinlock_profiling.cc:243] Waited 876 us on lock 0x7b180032aac0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49ac86d47 00007fe4a38dcb99 00007fe49aca369b 00007fe49aca193f 00007fe49aca1455 00007fe49dc11020 00007fe49dc236c6 00007fe49dc34237 00007fe49dc341b8
0905 20:00:20.802704 (+1552711us) log.cc:844] Serialized 14 byte log entry
0905 20:00:21.312523 (+509819us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:21.312630 (+ 107us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:21.312819 (+ 189us) raft_consensus.cc:1683] finished
0905 20:00:21.312833 (+ 14us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:21.313017 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":105132,"prepare.queue_time_us":571963,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":80,"raft.queue_time_us":168813,"raft.run_cpu_time_us":527,"raft.run_wall_time_us":18003,"spinlock_wait_cycles":2014080,"thread_start_us":1953556,"threads_started":4,"wal-append.queue_time_us":1161834,"wal-append.run_cpu_time_us":4617,"wal-append.run_wall_time_us":1513528}
I20250905 20:00:21.149766 31084 leader_election.cc:290] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:22.397918 27908 catalog_manager.cc:5582] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 1 to 5, leader changed from e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) to 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132). New cstate: current_term: 5 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 20:00:22.418593 28751 raft_consensus.cc:3058] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:22.433246 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 20:00:21.953387 31087 consensus_meta.cc:287] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.041s user 0.017s sys 0.001s
W20250905 20:00:21.653040 31083 consensus_meta.cc:287] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.591s user 0.008s sys 0.002s
I20250905 20:00:22.470461 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:22.393766 28081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50660 (request call id 342) took 1320 ms. Trace:
I20250905 20:00:22.471318 28081 rpcz_store.cc:276] 0905 20:00:21.073105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.073803 (+ 698us) service_pool.cc:225] Handling call
0905 20:00:22.393553 (+1319750us) spinlock_profiling.cc:243] Waited 15.7 ms on lock 0x7b540036db54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:22.393694 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":35997440}
W20250905 20:00:21.403635 31033 consensus_meta.cc:287] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.778s user 0.008s sys 0.007s
I20250905 20:00:22.438637 28009 rpcz_store.cc:276] 0905 20:00:21.091525 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.092182 (+ 657us) service_pool.cc:225] Handling call
0905 20:00:22.428766 (+1336584us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.437628 28005 rpcz_store.cc:276] 0905 20:00:21.268452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.268673 (+ 221us) service_pool.cc:225] Handling call
0905 20:00:22.426633 (+1157960us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.443187 28750 rpcz_store.cc:276] 0905 20:00:21.098008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.102289 (+ 4281us) service_pool.cc:225] Handling call
0905 20:00:21.116035 (+ 13746us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:22.380135 (+1264100us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:22.380339 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.406370 28082 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1637) took 1058 ms. Trace:
I20250905 20:00:22.419137 28004 raft_consensus.cc:2466] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
I20250905 20:00:22.420841 28085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1635) took 1074 ms. Trace:
I20250905 20:00:22.475248 28085 rpcz_store.cc:276] 0905 20:00:21.346863 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.348975 (+ 2112us) service_pool.cc:225] Handling call
0905 20:00:22.420764 (+1071789us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.444824 28002 rpcz_store.cc:276] 0905 20:00:21.375309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.375548 (+ 239us) service_pool.cc:225] Handling call
0905 20:00:21.641237 (+265689us) spinlock_profiling.cc:243] Waited 1.49 ms on lock 0x7b5800000f98. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b7db60 00007fe4a2b7d444 00007fe49c2050c4 00007fe49c205052
0905 20:00:22.444388 (+803151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3420032}
W20250905 20:00:21.775869 31162 log.cc:927] Time spent T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.155s user 0.001s sys 0.001s
I20250905 20:00:22.444557 28078 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1632) took 1370 ms. Trace:
I20250905 20:00:21.294853 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:22.449019 28745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40926 (request call id 531) took 1168 ms. Trace:
I20250905 20:00:22.478847 28745 rpcz_store.cc:276] 0905 20:00:21.280439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.281100 (+ 661us) service_pool.cc:225] Handling call
0905 20:00:22.448949 (+1167849us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.427395 28151 raft_consensus.cc:1238] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 3. Current term is 4. Ops: []
I20250905 20:00:22.447094 28746 raft_consensus.cc:2466] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 7.
I20250905 20:00:22.442719 28154 rpcz_store.cc:276] 0905 20:00:17.946359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.101693 (+3155334us) service_pool.cc:225] Handling call
0905 20:00:22.377680 (+1275987us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:22.378724 (+ 1044us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:22.398248 (+ 19524us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b5400326c50. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:22.398462 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2906728320}
I20250905 20:00:22.473924 28082 rpcz_store.cc:276] 0905 20:00:21.347558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.349555 (+ 1997us) service_pool.cc:225] Handling call
0905 20:00:21.550074 (+200519us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:22.391462 (+841388us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:22.406262 (+ 14800us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:21.745849 30978 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:22.494228 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:22.495086 28151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39586 (request call id 385) took 3365 ms. Trace:
I20250905 20:00:22.446985 30775 raft_consensus.cc:513] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:22.469571 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:22.498705 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:22.481257 28155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39586 (request call id 384) took 3350 ms. Trace:
I20250905 20:00:22.490813 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:22.482225 28156 raft_consensus.cc:2408] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20250905 20:00:21.102092 30884 raft_consensus.cc:513] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:22.478193 28078 rpcz_store.cc:276] 0905 20:00:21.073912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.094198 (+ 20286us) service_pool.cc:225] Handling call
0905 20:00:22.444490 (+1350292us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.470415 28157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39586 (request call id 381) took 3359 ms. Trace:
I20250905 20:00:22.468422 28150 raft_consensus.cc:3058] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:22.499457 28155 rpcz_store.cc:276] 0905 20:00:19.114898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.404883 (+3289985us) service_pool.cc:225] Handling call
0905 20:00:22.428940 (+ 24057us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:22.464647 (+ 35707us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:22.464999 (+ 352us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.500298 28157 rpcz_store.cc:276] 0905 20:00:19.097105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.425910 (+3328805us) service_pool.cc:225] Handling call
0905 20:00:22.426273 (+ 363us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:22.448397 (+ 22124us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:22.456653 (+ 8256us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.495553 28151 rpcz_store.cc:276] 0905 20:00:19.115244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.426809 (+3311565us) service_pool.cc:225] Handling call
0905 20:00:22.427139 (+ 330us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:22.480373 (+ 53234us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:22.480619 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.498368 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:21.671566 30849 consensus_meta.cc:287] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.583s user 0.012s sys 0.000s
W20250905 20:00:22.470017 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:22.507351 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:22.498072 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:22.502800 28158 raft_consensus.cc:2466] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
I20250905 20:00:22.509037 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:22.455499 28752 rpcz_store.cc:276] 0905 20:00:20.924690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:20.942840 (+ 18150us) service_pool.cc:225] Handling call
0905 20:00:20.984113 (+ 41273us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:22.376238 (+1392125us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:22.376283 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:22.376343 (+ 60us) raft_consensus.cc:1643] Marking committed up to 7
0905 20:00:22.407507 (+ 31164us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:22.428309 (+ 20802us) spinlock_profiling.cc:243] Waited 313 ms on lock 0x7b540053cd54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:22.428357 (+ 48us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:22.428494 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18121,"raft.queue_time_us":24018,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":141,"spinlock_wait_cycles":720464256}
I20250905 20:00:22.504273 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:22.506053 28081 raft_consensus.cc:2466] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:22.511620 28081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1645) took 1159 ms. Trace:
I20250905 20:00:22.509336 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:22.506609 28077 raft_consensus.cc:2391] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 6: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:22.492516 28001 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 11.
W20250905 20:00:22.499403 30777 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.952s user 0.010s sys 0.074s
I20250905 20:00:22.535522 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:22.530153 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:22.536484 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:22.529464 28010 raft_consensus.cc:2391] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 6: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:22.512394 28081 rpcz_store.cc:276] 0905 20:00:21.352568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.472454 (+1119886us) service_pool.cc:225] Handling call
0905 20:00:22.511519 (+ 39065us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 20:00:22.535254 30777 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.992s user 0.011s sys 0.074s
I20250905 20:00:22.541399 30884 leader_election.cc:290] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:22.542834 28159 raft_consensus.cc:2466] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 7.
I20250905 20:00:22.544739 28159 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39586 (request call id 386) took 3109 ms. Trace:
I20250905 20:00:22.545104 28159 rpcz_store.cc:276] 0905 20:00:19.435868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.543644 (+3107776us) service_pool.cc:225] Handling call
0905 20:00:22.544062 (+ 418us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:22.544374 (+ 312us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:22.544402 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:22.544449 (+ 47us) raft_consensus.cc:1643] Marking committed up to 9
0905 20:00:22.544490 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:22.544551 (+ 61us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:22.544662 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.547938 27908 catalog_manager.cc:5582] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 1 to 3, leader changed from e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130) to 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132). New cstate: current_term: 3 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 20:00:22.550868 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:22.552858 28158 raft_consensus.cc:1238] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 10 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 9. Current term is 10. Ops: []
I20250905 20:00:22.553529 28158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1703) took 2664 ms. Trace:
I20250905 20:00:22.554694 28085 raft_consensus.cc:2466] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 20:00:22.555383 28085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1671) took 1171 ms. Trace:
I20250905 20:00:22.555765 28085 rpcz_store.cc:276] 0905 20:00:21.384266 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.476285 (+1092019us) service_pool.cc:225] Handling call
0905 20:00:22.555316 (+ 79031us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.556926 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:22.557914 28085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1649) took 1204 ms. Trace:
I20250905 20:00:22.558764 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:22.559895 28752 raft_consensus.cc:2391] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 5: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 20:00:22.560421 28752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40926 (request call id 536) took 1212 ms. Trace:
I20250905 20:00:22.560748 28752 rpcz_store.cc:276] 0905 20:00:21.347946 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.510911 (+1162965us) service_pool.cc:225] Handling call
0905 20:00:22.560342 (+ 49431us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.562076 28748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40926 (request call id 535) took 1214 ms. Trace:
I20250905 20:00:22.562605 28152 raft_consensus.cc:2391] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 7: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:22.563695 28748 rpcz_store.cc:276] 0905 20:00:21.347626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.481528 (+1133902us) service_pool.cc:225] Handling call
0905 20:00:22.562000 (+ 80472us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.537389 28747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40926 (request call id 539) took 1188 ms. Trace:
I20250905 20:00:22.565063 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:22.565555 28747 rpcz_store.cc:276] 0905 20:00:21.348902 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.454969 (+1106067us) service_pool.cc:225] Handling call
0905 20:00:22.537309 (+ 82340us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.566051 28155 raft_consensus.cc:2466] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 7.
I20250905 20:00:22.535789 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:22.536145 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:22.567098 28749 raft_consensus.cc:3058] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:22.529995 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:22.558269 28085 rpcz_store.cc:276] 0905 20:00:21.353826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.556541 (+1202715us) service_pool.cc:225] Handling call
0905 20:00:22.557855 (+ 1314us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.553874 28158 rpcz_store.cc:276] 0905 20:00:19.889317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.552212 (+2662895us) service_pool.cc:225] Handling call
0905 20:00:22.552622 (+ 410us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:22.553275 (+ 653us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:22.553472 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.447018 30148 raft_consensus.cc:513] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:22.514919 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:22.572055 28078 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1675) took 1186 ms. Trace:
I20250905 20:00:22.574457 28082 raft_consensus.cc:2391] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 5: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 20:00:22.503783 30946 raft_consensus.cc:491] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:22.576439 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:22.577497 28745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40918 (request call id 632) took 1085 ms. Trace:
I20250905 20:00:22.577471 28078 rpcz_store.cc:276] 0905 20:00:21.385494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.500081 (+1114587us) service_pool.cc:225] Handling call
0905 20:00:22.571960 (+ 71879us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.578080 28157 raft_consensus.cc:3058] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:22.578341 28745 rpcz_store.cc:276] 0905 20:00:21.492246 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.498833 (+1006587us) service_pool.cc:225] Handling call
0905 20:00:22.577425 (+ 78592us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 20:00:22.533425 28077 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1651) took 1179 ms (client timeout 1554 ms). Trace:
I20250905 20:00:22.580039 28082 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1646) took 1227 ms. Trace:
W20250905 20:00:22.581650 28077 rpcz_store.cc:269] 0905 20:00:21.525668 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.462345 (+936677us) service_pool.cc:225] Handling call
0905 20:00:22.533317 (+ 70972us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.581780 28082 rpcz_store.cc:276] 0905 20:00:21.352859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.493738 (+1140879us) service_pool.cc:225] Handling call
0905 20:00:22.579962 (+ 86224us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.577021 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:22.583313 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:22.503619 30684 raft_consensus.cc:491] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:22.514142 30929 raft_consensus.cc:2747] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250905 20:00:22.587661 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:22.592893 28748 raft_consensus.cc:2391] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 7: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
W20250905 20:00:22.508010 28746 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40904 (request call id 744) took 1813 ms (client timeout 1912 ms). Trace:
I20250905 20:00:22.591219 28080 raft_consensus.cc:2391] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 5: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:22.542335 30884 raft_consensus.cc:2747] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:22.590011 28159 raft_consensus.cc:1273] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20250905 20:00:22.589507 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:22.595340 28081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1647) took 1242 ms. Trace:
I20250905 20:00:22.596472 28151 raft_consensus.cc:2408] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20250905 20:00:22.492630 31083 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:22.585215 30946 raft_consensus.cc:513] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:22.608258 28153 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2258) took 1780 ms. Trace:
I20250905 20:00:22.598608 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:22.599086 28748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40918 (request call id 635) took 1050 ms. Trace:
I20250905 20:00:22.502863 30849 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:22.575171 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 456036us
I20250905 20:00:22.601703 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:22.613829 28076 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Advancing to term 10
I20250905 20:00:22.537521 31031 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:22.529592 28743 raft_consensus.cc:2466] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 8.
I20250905 20:00:22.623198 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:22.604035 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:22.604424 28081 rpcz_store.cc:276] 0905 20:00:21.353163 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.573220 (+1220057us) service_pool.cc:225] Handling call
0905 20:00:22.595260 (+ 22040us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.624209 28077 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1674) took 1239 ms. Trace:
I20250905 20:00:22.602979 28159 pending_rounds.cc:85] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting all ops after (but not including) 3
W20250905 20:00:22.605933 28080 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1650) took 1247 ms (client timeout 1546 ms). Trace:
I20250905 20:00:22.624531 28077 rpcz_store.cc:276] 0905 20:00:21.385231 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.582805 (+1197574us) service_pool.cc:225] Handling call
0905 20:00:22.624141 (+ 41336us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.606473 28085 raft_consensus.cc:2408] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250905 20:00:22.608680 28153 rpcz_store.cc:276] 0905 20:00:20.845269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.574943 (+1729674us) service_pool.cc:225] Handling call
0905 20:00:22.585006 (+ 10063us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:22.607879 (+ 22873us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:22.607912 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:22.607970 (+ 58us) raft_consensus.cc:1643] Marking committed up to 7
0905 20:00:22.608004 (+ 34us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:22.608071 (+ 67us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:22.608197 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9845,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":101}
I20250905 20:00:22.609544 28151 raft_consensus.cc:1273] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
W20250905 20:00:22.624902 28080 rpcz_store.cc:269] 0905 20:00:21.354143 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.447684 (+1093541us) service_pool.cc:225] Handling call
0905 20:00:22.601188 (+153504us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 20:00:22.614840 28718 leader_election.cc:336] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.776s (SENT)
I20250905 20:00:22.611279 28748 rpcz_store.cc:276] 0905 20:00:21.547613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.564696 (+1017083us) service_pool.cc:225] Handling call
0905 20:00:22.597732 (+ 33036us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.609483 28158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2261) took 1762 ms. Trace:
I20250905 20:00:22.617244 28157 raft_consensus.cc:2466] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 7.
I20250905 20:00:22.618379 28156 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 2267) took 1722 ms. Trace:
I20250905 20:00:22.609901 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:22.609984 28747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40926 (request call id 538) took 1261 ms. Trace:
I20250905 20:00:22.603263 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:22.623580 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:22.612931 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:22.625572 28745 raft_consensus.cc:2466] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
I20250905 20:00:22.608690 27908 catalog_manager.cc:5582] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 20:00:22.626315 28151 pending_rounds.cc:85] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting all ops after (but not including) 2
I20250905 20:00:22.624737 28159 pending_rounds.cc:107] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20250905 20:00:22.628952 28158 rpcz_store.cc:276] 0905 20:00:20.846588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.576066 (+1729478us) service_pool.cc:225] Handling call
0905 20:00:22.607280 (+ 31214us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:22.608205 (+ 925us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:22.608342 (+ 137us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:22.608626 (+ 284us) raft_consensus.cc:1643] Marking committed up to 4
0905 20:00:22.608908 (+ 282us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:22.609061 (+ 153us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:22.609349 (+ 288us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.631143 28156 rpcz_store.cc:276] 0905 20:00:20.895967 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.048453 (+152486us) service_pool.cc:225] Handling call
0905 20:00:22.611303 (+1562850us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b54000eced4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:22.618289 (+ 6986us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3238219520}
I20250905 20:00:22.631537 28747 rpcz_store.cc:276] 0905 20:00:21.348593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.574708 (+1226115us) service_pool.cc:225] Handling call
0905 20:00:22.609913 (+ 35205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.634230 28085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1644) took 1282 ms. Trace:
I20250905 20:00:22.639636 28085 rpcz_store.cc:276] 0905 20:00:21.352266 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.568529 (+1216263us) service_pool.cc:225] Handling call
0905 20:00:22.634156 (+ 65627us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.634796 28159 raft_consensus.cc:2887] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250905 20:00:22.634672 28151 pending_rounds.cc:107] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20250905 20:00:22.633651 28750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40918 (request call id 636) took 1085 ms. Trace:
I20250905 20:00:22.633185 28745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40918 (request call id 637) took 1084 ms. Trace:
I20250905 20:00:22.641418 28151 raft_consensus.cc:2887] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250905 20:00:22.632915 28078 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1641) took 1281 ms. Trace:
I20250905 20:00:22.642108 28745 rpcz_store.cc:276] 0905 20:00:21.548975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.579071 (+1030096us) service_pool.cc:225] Handling call
0905 20:00:22.633116 (+ 54045us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.633415 28082 raft_consensus.cc:2466] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 20:00:22.608690 31083 leader_election.cc:290] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:22.642493 28078 rpcz_store.cc:276] 0905 20:00:21.351305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.578577 (+1227272us) service_pool.cc:225] Handling call
0905 20:00:22.632836 (+ 54259us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.643621 28082 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1642) took 1292 ms. Trace:
I20250905 20:00:22.641988 28750 rpcz_store.cc:276] 0905 20:00:21.548171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.495297 (+947126us) service_pool.cc:225] Handling call
0905 20:00:22.612754 (+117457us) spinlock_profiling.cc:243] Waited 5.23 ms on lock 0x7b5800304598. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b7db60 00007fe4a2b7d444 00007fe49c2050c4 00007fe49c205052
0905 20:00:22.633570 (+ 20816us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12033792}
I20250905 20:00:22.644006 28082 rpcz_store.cc:276] 0905 20:00:21.351635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.582907 (+1231272us) service_pool.cc:225] Handling call
0905 20:00:22.643551 (+ 60644us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.644773 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:22.645555 28748 raft_consensus.cc:2391] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 5: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 20:00:22.644835 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:22.646598 28157 raft_consensus.cc:2391] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 7: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:22.632251 28752 raft_consensus.cc:2466] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 1.
I20250905 20:00:22.648916 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:22.649343 28159 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1702) took 2777 ms. Trace:
I20250905 20:00:22.649544 28081 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Advancing to term 10
I20250905 20:00:22.649423 28752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40926 (request call id 537) took 1301 ms. Trace:
I20250905 20:00:22.649796 28159 rpcz_store.cc:276] 0905 20:00:19.872295 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.546091 (+2673796us) service_pool.cc:225] Handling call
0905 20:00:22.570544 (+ 24453us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:22.649086 (+ 78542us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:22.649285 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.650166 28752 rpcz_store.cc:276] 0905 20:00:21.348276 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.579227 (+1230951us) service_pool.cc:225] Handling call
0905 20:00:22.649268 (+ 70041us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.652489 28084 raft_consensus.cc:2466] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:22.653134 28084 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1638) took 1303 ms. Trace:
I20250905 20:00:22.653470 28084 rpcz_store.cc:276] 0905 20:00:21.350239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.356223 (+ 5984us) service_pool.cc:225] Handling call
0905 20:00:21.558199 (+201976us) spinlock_profiling.cc:243] Waited 42.4 ms on lock 0x7b5800204f98. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b7db60 00007fe4a2b7d444 00007fe49c2050c4 00007fe49c205052
0905 20:00:22.653068 (+1094869us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":97431296}
I20250905 20:00:22.584978 30684 raft_consensus.cc:513] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:22.655050 28151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2266) took 1761 ms. Trace:
I20250905 20:00:22.658421 28153 raft_consensus.cc:1238] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:22.658814 28151 rpcz_store.cc:276] 0905 20:00:20.895515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.604271 (+1708756us) service_pool.cc:225] Handling call
0905 20:00:22.609108 (+ 4837us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:22.654790 (+ 45682us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:22.654995 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 20:00:22.600977 28746 rpcz_store.cc:269] 0905 20:00:20.691779 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:20.731394 (+ 39615us) service_pool.cc:225] Handling call
0905 20:00:22.504909 (+1773515us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.661316 28154 raft_consensus.cc:2391] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 5: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
W20250905 20:00:22.634215 28743 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40926 (request call id 532) took 1353 ms (client timeout 1717 ms). Trace:
I20250905 20:00:22.590983 30978 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:22.662891 28156 raft_consensus.cc:1273] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20250905 20:00:22.663199 28159 raft_consensus.cc:1238] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:22.664465 28152 raft_consensus.cc:1273] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250905 20:00:22.670321 28152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2265) took 1822 ms. Trace:
I20250905 20:00:22.670775 28152 rpcz_store.cc:276] 0905 20:00:20.863020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.577724 (+1714704us) service_pool.cc:225] Handling call
0905 20:00:22.635468 (+ 57744us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:22.670109 (+ 34641us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:22.670268 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.667389 28077 raft_consensus.cc:1273] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20250905 20:00:22.664228 28001 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 2145) took 1288 ms (client timeout 1514 ms). Trace:
I20250905 20:00:22.673527 28077 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1639) took 1322 ms. Trace:
I20250905 20:00:22.674005 28077 rpcz_store.cc:276] 0905 20:00:21.350621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.625362 (+1274741us) service_pool.cc:225] Handling call
0905 20:00:22.647737 (+ 22375us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:22.673062 (+ 25325us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:22.673460 (+ 398us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.669512 28156 pending_rounds.cc:85] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting all ops after (but not including) 3
I20250905 20:00:22.675522 28156 pending_rounds.cc:107] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20250905 20:00:22.675794 28156 raft_consensus.cc:2887] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250905 20:00:22.677166 28155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2260) took 1831 ms. Trace:
I20250905 20:00:22.677578 28155 rpcz_store.cc:276] 0905 20:00:20.846236 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.579313 (+1733077us) service_pool.cc:225] Handling call
0905 20:00:22.635724 (+ 56411us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:22.670619 (+ 34895us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:22.670646 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:22.676903 (+ 6257us) raft_consensus.cc:1643] Marking committed up to 6
0905 20:00:22.676945 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:22.677004 (+ 59us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:22.677110 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 20:00:22.679658 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 64874us
I20250905 20:00:22.631434 31031 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:22.686882 28151 raft_consensus.cc:1238] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:22.686904 28083 raft_consensus.cc:2466] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:22.687808 28083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1633) took 1428 ms. Trace:
W20250905 20:00:22.673110 28001 rpcz_store.cc:269] 0905 20:00:21.375761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.377682 (+ 1921us) service_pool.cc:225] Handling call
0905 20:00:22.664056 (+1286374us) spinlock_profiling.cc:243] Waited 50.7 ms on lock 0x7b54000546d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:22.664168 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":116701568}
I20250905 20:00:22.688155 28083 rpcz_store.cc:276] 0905 20:00:21.259846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.272351 (+ 12505us) service_pool.cc:225] Handling call
0905 20:00:22.687750 (+1415399us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.688709 28158 raft_consensus.cc:1238] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20250905 20:00:22.663199 30148 leader_election.cc:290] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:22.696087 28154 raft_consensus.cc:1273] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20250905 20:00:22.696559 28154 pending_rounds.cc:85] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting all ops after (but not including) 5
I20250905 20:00:22.697731 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:22.702561 28154 pending_rounds.cc:107] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20250905 20:00:22.703104 28154 raft_consensus.cc:2887] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250905 20:00:22.390787 27904 rpcz_store.cc:269] 0905 19:59:53.215229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:59:53.348832 (+133603us) service_pool.cc:225] Handling call
0905 20:00:21.620867 (+28272035us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 20:00:11.521885 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 20:00:11.522653 (+ 768us) write_op.cc:432] Acquiring schema lock in shared mode
0905 20:00:11.522700 (+ 47us) write_op.cc:435] Acquired schema lock
0905 20:00:11.522719 (+ 19us) tablet.cc:660] Decoding operations
0905 20:00:11.523762 (+ 1043us) write_op.cc:620] Acquiring the partition lock for write op
0905 20:00:11.523903 (+ 141us) write_op.cc:641] Partition lock acquired for write op
0905 20:00:11.523946 (+ 43us) tablet.cc:683] Acquiring locks for 6 operations
0905 20:00:11.524849 (+ 903us) tablet.cc:699] Row locks acquired
0905 20:00:11.524872 (+ 23us) write_op.cc:260] PREPARE: finished
0905 20:00:11.525158 (+ 286us) write_op.cc:270] Start()
0905 20:00:11.525375 (+ 217us) write_op.cc:276] Timestamp: P: 1757102411525046 usec, L: 0
0905 20:00:11.525400 (+ 25us) op_driver.cc:348] REPLICATION: starting
0905 20:00:11.535281 (+ 9881us) log.cc:844] Serialized 2468 byte log entry
0905 20:00:12.167632 (+632351us) op_driver.cc:464] REPLICATION: finished
0905 20:00:13.479133 (+1311501us) spinlock_profiling.cc:243] Waited 1.87 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49ac86fec 00007fe4a38dcb99 00007fe49aca369b 00007fe49aca193f 00007fe49aca3c02 00007fe4a1643e60
0905 20:00:13.628579 (+149446us) write_op.cc:301] APPLY: starting
0905 20:00:13.649691 (+ 21112us) tablet.cc:1365] starting BulkCheckPresence
0905 20:00:13.758108 (+108417us) tablet.cc:1368] finished BulkCheckPresence
0905 20:00:13.781783 (+ 23675us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 20:00:14.127736 (+345953us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 20:00:14.416374 (+288638us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0905 20:00:14.479228 (+ 62854us) write_op.cc:312] APPLY: finished
0905 20:00:14.520948 (+ 41720us) log.cc:844] Serialized 53 byte log entry
0905 20:00:15.776475 (+1255527us) spinlock_profiling.cc:243] Waited 47.1 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49ac86fec 00007fe4a38dcb99 00007fe49aca369b 00007fe49aca193f 00007fe49aca3c02 00007fe49d8572bb
0905 20:00:16.941516 (+1165041us) write_op.cc:489] Releasing partition, row and schema locks
0905 20:00:16.960863 (+ 19347us) write_op.cc:454] Released schema lock
0905 20:00:16.968743 (+ 7880us) write_op.cc:341] FINISH: Updating metrics
0905 20:00:16.993057 (+ 24314us) write_op.cc:489] Releasing partition, row and schema locks
0905 20:00:17.000981 (+ 7924us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":50963,"reported_tablets":31,"child_traces":[["op",{"apply.queue_time_us":1417565,"apply.run_cpu_time_us":45559,"apply.run_wall_time_us":3362905,"num_ops":6,"prepare.queue_time_us":201561,"prepare.run_cpu_time_us":18916,"prepare.run_wall_time_us":199245,"raft.queue_time_us":259339,"raft.run_cpu_time_us":32744,"raft.run_wall_time_us":1513757,"replication_time_us":610602,"spinlock_wait_cycles":112672768,"thread_start_us":4221411,"threads_started":5,"wal-append.queue_time_us":2203661,"wal-append.run_cpu_time_us":16143,"wal-append.run_wall_time_us":2168910}]]}
I20250905 20:00:22.714643 28153 raft_consensus.cc:2466] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 8.
I20250905 20:00:22.690143 31093 raft_consensus.cc:491] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:22.666082 28743 rpcz_store.cc:269] 0905 20:00:21.280934 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.281653 (+ 719us) service_pool.cc:225] Handling call
0905 20:00:22.630751 (+1349098us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.732051 28150 raft_consensus.cc:2466] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
I20250905 20:00:22.753574 28079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 341) took 1681 ms. Trace:
I20250905 20:00:22.757113 28079 rpcz_store.cc:276] 0905 20:00:21.072643 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.103034 (+ 30391us) service_pool.cc:225] Handling call
0905 20:00:21.343011 (+239977us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:21.343902 (+ 891us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:21.343942 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:21.344689 (+ 747us) log.cc:844] Serialized 26 byte log entry
0905 20:00:21.592847 (+248158us) raft_consensus.cc:1643] Marking committed up to 7
0905 20:00:21.592921 (+ 74us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:21.593028 (+ 107us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:22.753267 (+1160239us) raft_consensus.cc:1683] finished
0905 20:00:22.753293 (+ 26us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:22.753471 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1861,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":120,"thread_start_us":247482,"threads_started":1,"wal-append.queue_time_us":1357262}
I20250905 20:00:22.743137 31087 raft_consensus.cc:491] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:22.722445 31093 raft_consensus.cc:513] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:22.754292 30775 leader_election.cc:290] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:22.779592 30946 leader_election.cc:290] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:22.783771 28081 raft_consensus.cc:2408] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20250905 20:00:22.783581 28751 raft_consensus.cc:2466] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 8.
I20250905 20:00:22.787958 28750 raft_consensus.cc:1238] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 7. Current term is 8. Ops: []
I20250905 20:00:22.788187 28081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1673) took 1403 ms. Trace:
W20250905 20:00:22.787510 28751 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40926 (request call id 530) took 1545 ms (client timeout 1691 ms). Trace:
I20250905 20:00:22.788673 28081 rpcz_store.cc:276] 0905 20:00:21.384932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.624533 (+1239601us) service_pool.cc:225] Handling call
0905 20:00:22.788115 (+163582us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.792062 28750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40918 (request call id 634) took 1244 ms. Trace:
W20250905 20:00:22.792606 31164 log.cc:927] Time spent T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.117s user 0.000s sys 0.001s
I20250905 20:00:22.792609 28750 rpcz_store.cc:276] 0905 20:00:21.546955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.644457 (+1097502us) service_pool.cc:225] Handling call
0905 20:00:22.787427 (+142970us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:22.788589 (+ 1162us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:22.790782 (+ 2193us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b5400432950. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:22.791015 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":328085376}
W20250905 20:00:22.789911 28751 rpcz_store.cc:269] 0905 20:00:21.242216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.242452 (+ 236us) service_pool.cc:225] Handling call
0905 20:00:22.787444 (+1544992us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.779675 31099 leader_election.cc:290] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:22.798605 28154 raft_consensus.cc:1238] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:22.790721 30849 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:22.814332 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:22.816416 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:22.782420 31087 raft_consensus.cc:513] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:22.833731 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:22.823140 30684 leader_election.cc:290] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:22.811028 31151 raft_consensus.cc:491] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:22.843861 28154 raft_consensus.cc:2391] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 8: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
W20250905 20:00:22.846290 31161 log.cc:927] Time spent T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.165s user 0.001s sys 0.001s
W20250905 20:00:22.847410 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:22.835566 31031 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:22.843246 31151 raft_consensus.cc:513] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:22.851405 30978 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:22.851824 31093 leader_election.cc:290] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:22.852273 28152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:33742 (request call id 724) took 3808 ms. Trace:
I20250905 20:00:22.852793 31151 leader_election.cc:290] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:22.852622 31093 consensus_queue.cc:1035] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 20:00:22.852669 28152 rpcz_store.cc:276] 0905 20:00:19.046688 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.683133 (+3636445us) service_pool.cc:225] Handling call
0905 20:00:22.852155 (+169022us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:22.866941 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:22.867785 28154 raft_consensus.cc:2466] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 7.
I20250905 20:00:22.875319 28076 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 10.
I20250905 20:00:22.874265 28749 raft_consensus.cc:2466] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 8.
W20250905 20:00:22.876494 28076 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1648) took 1522 ms (client timeout 1717 ms). Trace:
I20250905 20:00:22.876590 28747 raft_consensus.cc:1238] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 7. Current term is 8. Ops: []
W20250905 20:00:22.876997 28076 rpcz_store.cc:269] 0905 20:00:21.353483 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.465647 (+1112164us) service_pool.cc:225] Handling call
0905 20:00:22.876170 (+410523us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 20:00:22.890295 28715 connection.cc:232] RPC call timeout handler was delayed by -1.7118847599999754s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1736s for a response
W20250905 20:00:22.891440 28715 connection.cc:232] RPC call timeout handler was delayed by -1.2582847600000377s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22400000000000003s for a response
W20250905 20:00:22.919924 28749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40918 (request call id 638) took 1340 ms (client timeout 1699 ms). Trace:
W20250905 20:00:22.920634 28749 rpcz_store.cc:269] 0905 20:00:21.663130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.503424 (+840294us) service_pool.cc:225] Handling call
0905 20:00:22.890202 (+386778us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 20:00:22.935479 27973 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.236s user 0.010s sys 0.017s
W20250905 20:00:22.936969 27973 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.238s user 0.010s sys 0.018s
W20250905 20:00:22.934959 27976 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.021s sys 0.004s
W20250905 20:00:22.939091 27976 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.521s user 0.021s sys 0.005s
W20250905 20:00:22.939668 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:22.943215 30978 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:22.944108 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:22.952373 28744 raft_consensus.cc:2466] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
W20250905 20:00:22.953665 28744 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40918 (request call id 631) took 1788 ms (client timeout 1729 ms). Trace:
W20250905 20:00:22.954211 28744 rpcz_store.cc:269] 0905 20:00:21.168106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:21.240626 (+ 72520us) service_pool.cc:225] Handling call
0905 20:00:22.953557 (+1712931us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 20:00:22.958539 27974 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.007s sys 0.019s
W20250905 20:00:22.962019 27974 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.556s user 0.007s sys 0.020s
W20250905 20:00:22.964365 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:22.962790 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 567106us
W20250905 20:00:22.971604 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:22.977094 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:22.955049 30978 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:22.976886 28050 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.010s sys 0.011s
W20250905 20:00:22.979012 28050 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.533s user 0.010s sys 0.011s
W20250905 20:00:22.976121 28123 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.010s sys 0.010s
W20250905 20:00:22.980175 28123 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.511s user 0.010s sys 0.011s
W20250905 20:00:22.979501 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 561781us
W20250905 20:00:22.978260 28051 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.006s sys 0.015s
W20250905 20:00:22.982276 28051 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.537s user 0.006s sys 0.015s
I20250905 20:00:22.982147 28001 raft_consensus.cc:2366] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 6. Current term is 7.
W20250905 20:00:22.982964 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 554966us
I20250905 20:00:22.986209 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:22.988965 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 454590us
W20250905 20:00:22.988883 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 495967us
I20250905 20:00:22.984122 30684 raft_consensus.cc:491] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:22.993739 28747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40926 (request call id 534) took 1646 ms. Trace:
I20250905 20:00:22.994375 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:22.995544 28001 raft_consensus.cc:2466] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:22.995210 28747 rpcz_store.cc:276] 0905 20:00:21.347339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.637242 (+1289903us) service_pool.cc:225] Handling call
0905 20:00:22.876337 (+239095us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:22.877129 (+ 792us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:22.993561 (+116432us) spinlock_profiling.cc:243] Waited 220 ms on lock 0x7b540062b1d0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:22.993680 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":506573056}
I20250905 20:00:22.997031 30684 raft_consensus.cc:513] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:22.998916 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:22.998534 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 1.87493e+06us
I20250905 20:00:23.000819 30978 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:22.990939 28010 raft_consensus.cc:2466] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
I20250905 20:00:23.008481 28002 raft_consensus.cc:2466] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 9.
I20250905 20:00:22.990571 30849 raft_consensus.cc:491] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:23.010814 27975 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:23.011328 27975 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:23.007956 28717 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.908s user 0.006s sys 0.022s
W20250905 20:00:23.013167 28717 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.985s user 0.006s sys 0.022s
W20250905 20:00:23.013360 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 1.02831e+06us
W20250905 20:00:23.016826 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:23.017398 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:23.018190 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:23.018602 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:23.018877 28744 raft_consensus.cc:3058] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:23.018848 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:23.019831 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:23.020459 28004 raft_consensus.cc:2466] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
I20250905 20:00:23.021957 30849 raft_consensus.cc:513] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:23.025787 27974 leader_election.cc:304] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 20:00:23.032027 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:23.032792 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:23.033238 28123 connection.cc:232] RPC call timeout handler was delayed by -1.6583061689999568s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21330000000000002s for a response
I20250905 20:00:23.034035 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:23.035887 28002 raft_consensus.cc:1238] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 5. Current term is 7. Ops: []
W20250905 20:00:23.037278 27975 connection.cc:232] RPC call timeout handler was delayed by -1.1876093320000791s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1564s for a response
W20250905 20:00:23.037678 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 1.70718e+06us
I20250905 20:00:23.040752 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:23.041577 28153 raft_consensus.cc:2466] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 20:00:23.038484 28051 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:23.042886 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:23.043236 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:23.043907 28751 raft_consensus.cc:2466] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
I20250905 20:00:23.045543 28747 raft_consensus.cc:1238] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 3. Current term is 4. Ops: []
I20250905 20:00:23.024461 30849 leader_election.cc:290] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:23.047616 28123 connection.cc:232] RPC call timeout handler was delayed by -1.1313319250000404s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1531s for a response
I20250905 20:00:23.022308 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:23.051344 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:23.050684 28154 raft_consensus.cc:2466] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 1.
I20250905 20:00:23.052475 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:23.054076 31177 raft_consensus.cc:491] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:23.054348 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:23.054276 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:23.055518 28156 raft_consensus.cc:2466] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 8.
I20250905 20:00:23.056747 27974 leader_election.cc:304] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:23.057917 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:23.057696 28050 leader_election.cc:304] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:23.062502 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:23.066896 31175 raft_consensus.cc:2747] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250905 20:00:23.072039 28158 raft_consensus.cc:2408] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20250905 20:00:23.072643 28750 raft_consensus.cc:2466] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 8.
I20250905 20:00:23.055204 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:23.071445 30978 raft_consensus.cc:2802] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:23.073590 31084 raft_consensus.cc:2802] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader election won for term 7
I20250905 20:00:23.076138 27974 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:23.057642 31099 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:23.073858 31174 raft_consensus.cc:491] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:23.078693 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:23.079228 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 20:00:23.080350 28122 connection.cc:232] RPC call timeout handler was delayed by -2.1492611720000241s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1658s for a response
W20250905 20:00:23.072613 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:23.083356 28004 raft_consensus.cc:3058] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
W20250905 20:00:23.083407 28122 connection.cc:232] RPC call timeout handler was delayed by -1.8711611720000292s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1967s for a response
W20250905 20:00:23.083910 28122 connection.cc:232] RPC call timeout handler was delayed by -1.7865611720000061s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2061s for a response
W20250905 20:00:23.084357 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:23.086112 28744 raft_consensus.cc:2466] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 8.
I20250905 20:00:23.078016 31172 consensus_queue.cc:1046] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20250905 20:00:23.086930 28746 raft_consensus.cc:2391] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 10: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 20:00:23.063313 31177 raft_consensus.cc:513] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:23.089852 28122 connection.cc:232] RPC call timeout handler was delayed by -1.721761172000015s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21330000000000002s for a response
W20250905 20:00:23.092125 28122 connection.cc:232] RPC call timeout handler was delayed by -1.3923611720000508s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2499s for a response
I20250905 20:00:23.094169 27974 leader_election.cc:304] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:23.094341 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:23.090274 31099 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:23.095818 31087 leader_election.cc:290] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:23.096374 28050 leader_election.cc:304] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:23.077306 30978 raft_consensus.cc:491] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:23.082966 31093 raft_consensus.cc:2802] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 20:00:23.097851 28157 raft_consensus.cc:1273] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250905 20:00:23.099725 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:23.090468 30775 raft_consensus.cc:2802] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader election won for term 8
I20250905 20:00:23.098872 31084 raft_consensus.cc:695] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:23.103164 28010 raft_consensus.cc:2466] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
W20250905 20:00:23.103470 31159 log.cc:927] Time spent T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.319s user 0.007s sys 0.000s
I20250905 20:00:23.090863 31174 raft_consensus.cc:513] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:23.082170 31176 consensus_queue.cc:1035] T 50ac9970ddf042fda15f38b623aa7dd0 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.088s
W20250905 20:00:23.105266 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 89601.7us
I20250905 20:00:23.079173 30849 consensus_queue.cc:1046] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20250905 20:00:23.099135 30978 raft_consensus.cc:3058] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:23.110540 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:23.110672 28123 leader_election.cc:304] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:23.104841 30777 raft_consensus.cc:2802] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:23.104238 31093 raft_consensus.cc:491] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:23.109380 28158 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
from {username='slave'} at 127.0.0.1:36164
I20250905 20:00:23.109381 31174 leader_election.cc:290] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:23.121613 28158 raft_consensus.cc:604] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Received request to transfer leadership to TS e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:23.122123 28158 raft_consensus.cc:612] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250905 20:00:23.113915 28010 raft_consensus.cc:2374] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6. Re-sending same reply.
I20250905 20:00:23.112958 27975 leader_election.cc:304] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:23.109388 30775 raft_consensus.cc:695] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:23.125352 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:23.107604 31084 consensus_queue.cc:237] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:23.124692 31178 raft_consensus.cc:2802] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:23.127357 31178 raft_consensus.cc:491] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:23.127777 31178 raft_consensus.cc:3058] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:23.128376 30148 raft_consensus.cc:3053] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 20:00:23.128810 30148 raft_consensus.cc:738] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:23.123030 31172 raft_consensus.cc:2802] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader election won for term 8
I20250905 20:00:23.138178 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:23.113837 30777 raft_consensus.cc:491] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:23.140039 30777 raft_consensus.cc:3058] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:23.139180 28001 raft_consensus.cc:3058] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
W20250905 20:00:23.143733 28718 leader_election.cc:336] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.793s (SENT)
I20250905 20:00:23.121869 31093 raft_consensus.cc:3058] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:23.148885 28123 leader_election.cc:304] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:23.123808 30684 leader_election.cc:290] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:23.137208 31171 raft_consensus.cc:3053] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Stepping down as leader of term 4
I20250905 20:00:23.150403 31171 raft_consensus.cc:738] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:23.150561 31177 leader_election.cc:290] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:23.147957 31172 raft_consensus.cc:695] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:23.151366 31171 consensus_queue.cc:260] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:23.151798 31172 consensus_queue.cc:237] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:23.152983 31171 raft_consensus.cc:3058] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:23.150449 30946 consensus_queue.cc:1046] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20250905 20:00:23.155951 31099 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:23.155603 30775 consensus_queue.cc:237] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:23.157449 28010 raft_consensus.cc:1238] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 10 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 9. Current term is 10. Ops: []
I20250905 20:00:23.149569 30849 raft_consensus.cc:2802] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader election won for term 7
I20250905 20:00:23.129855 30148 consensus_queue.cc:260] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:23.182866 28001 raft_consensus.cc:2466] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:23.185060 28158 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "6db33851e9914ca58560e8902701dca3"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
from {username='slave'} at 127.0.0.1:36164
I20250905 20:00:23.205811 28158 raft_consensus.cc:604] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 LEADER]: Received request to transfer leadership to TS e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:23.190877 30148 raft_consensus.cc:3058] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:23.193250 28085 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1672) took 1808 ms. Trace:
I20250905 20:00:23.165243 30849 raft_consensus.cc:695] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:23.178160 30978 raft_consensus.cc:513] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:23.199867 28080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1670) took 1816 ms. Trace:
I20250905 20:00:23.210927 28051 leader_election.cc:304] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:23.214926 28004 raft_consensus.cc:2466] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
W20250905 20:00:23.216302 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 51195us
I20250905 20:00:23.219146 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:23.222204 28001 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 20:00:23.214380 28085 rpcz_store.cc:276] 0905 20:00:21.384593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.640613 (+1256020us) service_pool.cc:225] Handling call
0905 20:00:22.655346 (+ 14733us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:22.663284 (+ 7938us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:22.663311 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:22.702943 (+ 39632us) log.cc:844] Serialized 26 byte log entry
0905 20:00:23.069107 (+366164us) raft_consensus.cc:1643] Marking committed up to 4
0905 20:00:23.069138 (+ 31us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:23.069192 (+ 54us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:23.193010 (+123818us) raft_consensus.cc:1683] finished
0905 20:00:23.193030 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:23.193186 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":308076,"threads_started":1,"wal-append.queue_time_us":391584}
W20250905 20:00:23.223146 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 53196.2us
I20250905 20:00:23.227114 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:23.227566 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:23.228521 28003 raft_consensus.cc:2408] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250905 20:00:23.228592 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:23.236711 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:23.237741 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:23.238834 28005 raft_consensus.cc:2391] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 6: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:23.251900 28009 raft_consensus.cc:1273] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250905 20:00:23.221837 31177 raft_consensus.cc:2802] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 20:00:23.219887 28080 rpcz_store.cc:276] 0905 20:00:21.383848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:22.627141 (+1243293us) service_pool.cc:225] Handling call
0905 20:00:22.652590 (+ 25449us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:22.682250 (+ 29660us) raft_consensus.cc:1532] Early marking committed up to index 3
0905 20:00:22.726098 (+ 43848us) log.cc:844] Serialized 14 byte log entry
0905 20:00:23.199296 (+473198us) log.cc:844] Serialized 14 byte log entry
0905 20:00:23.199531 (+ 235us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:23.199575 (+ 44us) raft_consensus.cc:1643] Marking committed up to 3
0905 20:00:23.199619 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:23.199683 (+ 64us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:23.199807 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2817,"prepare.queue_time_us":38475,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":2244,"raft.queue_time_us":13282,"raft.run_cpu_time_us":304,"raft.run_wall_time_us":12853,"thread_start_us":408800,"threads_started":2,"wal-append.queue_time_us":348873}
I20250905 20:00:23.252672 31175 raft_consensus.cc:2802] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250905 20:00:23.255779 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 83121.3us
I20250905 20:00:23.258806 28051 leader_election.cc:304] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:23.261804 28050 leader_election.cc:304] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:23.254288 31177 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:23.274932 31177 raft_consensus.cc:3058] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:23.275933 30777 raft_consensus.cc:513] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:23.287158 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:23.280747 31187 raft_consensus.cc:2747] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250905 20:00:23.282024 31151 consensus_queue.cc:1035] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20250905 20:00:23.287997 28009 raft_consensus.cc:2366] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 6. Current term is 7.
I20250905 20:00:23.290169 28049 leader_election.cc:304] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:23.292001 28050 leader_election.cc:304] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:23.292222 31196 raft_consensus.cc:2802] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election won for term 6
I20250905 20:00:23.294667 31189 consensus_queue.cc:1035] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 20:00:23.258296 31175 raft_consensus.cc:491] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
W20250905 20:00:23.301481 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 162739us
W20250905 20:00:23.307708 28125 connection.cc:232] RPC call timeout handler was delayed by -2.414532912000027s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1882s for a response
I20250905 20:00:23.310602 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:23.311403 28006 raft_consensus.cc:2466] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:23.312850 30777 leader_election.cc:290] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:23.313304 31189 raft_consensus.cc:2802] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 20:00:23.313840 31189 raft_consensus.cc:491] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:23.313881 30777 raft_consensus.cc:2802] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election won for term 6
I20250905 20:00:23.314328 31189 raft_consensus.cc:3058] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 3 FOLLOWER]: Advancing to term 4
I20250905 20:00:23.314788 30777 raft_consensus.cc:695] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:23.315834 30777 consensus_queue.cc:237] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:23.310020 30978 leader_election.cc:290] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:23.304307 31175 raft_consensus.cc:3058] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:23.346992 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:23.301126 30849 consensus_queue.cc:237] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:23.344126 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:23.348037 28001 raft_consensus.cc:2466] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:23.346156 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:23.348780 28003 raft_consensus.cc:2408] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250905 20:00:23.346621 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:23.336403 30978 consensus_queue.cc:1046] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
W20250905 20:00:23.330567 31182 log.cc:927] Time spent T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.211s user 0.002s sys 0.000s
W20250905 20:00:23.345566 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:23.336724 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:23.336093 27975 leader_election.cc:304] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:23.339169 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:23.346601 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:23.337795 28006 raft_consensus.cc:1273] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250905 20:00:23.353497 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:23.343215 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:23.351969 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:23.355204 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:23.355655 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:23.339406 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:23.359989 27975 leader_election.cc:304] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a
W20250905 20:00:23.355209 28125 leader_election.cc:336] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.605s (SENT)
I20250905 20:00:23.361620 28125 leader_election.cc:304] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
W20250905 20:00:23.361927 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 59187.4us
W20250905 20:00:23.363447 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 73321us
W20250905 20:00:23.364264 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:23.365494 28049 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 20:00:23.367983 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:23.374006 30148 raft_consensus.cc:2802] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 20:00:23.374552 30148 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:23.374930 30148 raft_consensus.cc:3058] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:23.377974 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 20:00:23.368325 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 76433.1us
I20250905 20:00:23.381810 28050 leader_election.cc:304] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 20:00:23.382944 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:23.386101 27975 leader_election.cc:304] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:23.389806 30148 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:23.336998 31151 consensus_queue.cc:1035] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.055s
I20250905 20:00:23.393543 31151 consensus_queue.cc:1046] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250905 20:00:23.325428 31187 consensus_queue.cc:1035] T a3efc6666ba5411c9b50c9d6e34089ad P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.050s
I20250905 20:00:23.395989 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:23.384382 31174 raft_consensus.cc:2802] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 20:00:23.370164 31093 raft_consensus.cc:513] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:23.398213 31174 raft_consensus.cc:491] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:23.399916 31084 consensus_queue.cc:1046] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250905 20:00:23.400382 31174 raft_consensus.cc:3058] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:23.400960 31084 raft_consensus.cc:3053] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 LEADER]: Stepping down as leader of term 9
I20250905 20:00:23.401364 31084 raft_consensus.cc:738] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:23.398064 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:23.402650 31195 raft_consensus.cc:2747] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250905 20:00:23.403628 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 48470us
I20250905 20:00:23.404062 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:23.404920 28743 raft_consensus.cc:2374] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4. Re-sending same reply.
W20250905 20:00:23.405697 28125 connection.cc:232] RPC call timeout handler was delayed by -2.2534329119999938s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2061s for a response
I20250905 20:00:23.405886 31190 raft_consensus.cc:3053] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 LEADER]: Stepping down as leader of term 6
W20250905 20:00:23.398957 28718 leader_election.cc:336] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 2.035s (SENT)
I20250905 20:00:23.406459 31190 raft_consensus.cc:738] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:23.407640 31190 consensus_queue.cc:260] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:23.409390 31190 raft_consensus.cc:3058] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:23.414196 30148 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:23.420387 28049 leader_election.cc:304] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 20:00:23.421886 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 54691.4us
W20250905 20:00:23.422094 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 138743us
I20250905 20:00:23.403903 31093 leader_election.cc:290] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:23.425568 28007 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 20:00:23.414191 30684 raft_consensus.cc:2802] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader election won for term 7
W20250905 20:00:23.428217 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 46545.6us
I20250905 20:00:23.414012 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:23.336496 31178 raft_consensus.cc:513] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:23.432366 28123 leader_election.cc:304] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:23.429448 28050 leader_election.cc:304] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:23.433452 30148 consensus_queue.cc:1035] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20250905 20:00:23.434414 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 79089.4us
W20250905 20:00:23.448836 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:23.435150 28005 raft_consensus.cc:1238] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 LEADER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20250905 20:00:23.454547 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:23.423918 31196 raft_consensus.cc:695] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:23.457888 28051 leader_election.cc:304] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:23.452807 30775 raft_consensus.cc:3053] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Stepping down as leader of term 4
I20250905 20:00:23.470749 31198 raft_consensus.cc:2802] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader election won for term 8
W20250905 20:00:23.496443 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 44385.6us
I20250905 20:00:23.474388 31187 raft_consensus.cc:2802] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 20:00:23.451387 31172 consensus_queue.cc:1046] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20250905 20:00:23.423399 31084 consensus_queue.cc:260] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:23.474720 31203 raft_consensus.cc:3053] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 LEADER]: Stepping down as leader of term 6
I20250905 20:00:23.471379 31197 raft_consensus.cc:2802] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election won for term 5
W20250905 20:00:23.496675 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 64449.1us
W20250905 20:00:23.499320 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 69995.9us
W20250905 20:00:23.498240 28125 connection.cc:232] RPC call timeout handler was delayed by -2.2334113279999883s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17270000000000002s for a response
I20250905 20:00:23.508756 31196 consensus_queue.cc:237] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:23.510862 31198 raft_consensus.cc:695] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [term 8 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
W20250905 20:00:23.511129 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 80875.8us
I20250905 20:00:23.453699 30684 raft_consensus.cc:695] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
W20250905 20:00:23.526259 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 105941us
I20250905 20:00:23.495899 30775 raft_consensus.cc:738] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:23.526381 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:23.530527 28718 leader_election.cc:336] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 2.180s (SENT)
I20250905 20:00:23.531654 31093 raft_consensus.cc:2802] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:23.478253 30862 consensus_queue.cc:1046] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20250905 20:00:23.499859 31203 raft_consensus.cc:738] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:23.499408 31195 raft_consensus.cc:2802] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 20:00:23.536216 28718 leader_election.cc:304] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:23.540771 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:23.566082 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:23.570755 28007 raft_consensus.cc:2466] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
W20250905 20:00:23.573484 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:23.574147 28125 connection.cc:232] RPC call timeout handler was delayed by -2.0692836869999383s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1967s for a response
W20250905 20:00:23.574649 28125 connection.cc:232] RPC call timeout handler was delayed by -1.8592329120000386s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2499s for a response
W20250905 20:00:23.575279 28125 connection.cc:232] RPC call timeout handler was delayed by -1.5926113279999754s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2439s for a response
W20250905 20:00:23.575762 28125 connection.cc:232] RPC call timeout handler was delayed by -1.0930924409999534s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1503s for a response
W20250905 20:00:23.576263 28125 connection.cc:232] RPC call timeout handler was delayed by -1.0807329120000304s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33640000000000003s for a response
W20250905 20:00:23.576711 28125 connection.cc:232] RPC call timeout handler was delayed by -1.0417924409999841s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15600000000000003s for a response
I20250905 20:00:23.534152 31093 raft_consensus.cc:491] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:23.579110 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:23.581168 28152 raft_consensus.cc:2466] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 20:00:23.581311 27975 leader_election.cc:304] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:23.580482 30148 consensus_queue.cc:1035] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20250905 20:00:23.585366 28158 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:36164
I20250905 20:00:23.588877 28158 raft_consensus.cc:604] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Received request to transfer leadership to TS cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:23.589293 28158 raft_consensus.cc:612] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250905 20:00:23.586730 28005 raft_consensus.cc:1273] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20250905 20:00:23.589918 28005 pending_rounds.cc:85] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Aborting all ops after (but not including) 3
I20250905 20:00:23.590121 28005 pending_rounds.cc:107] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20250905 20:00:23.536075 31200 raft_consensus.cc:2802] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 20:00:23.590328 28005 raft_consensus.cc:2887] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250905 20:00:23.548046 31195 raft_consensus.cc:491] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:23.595422 28051 leader_election.cc:304] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 20:00:23.595103 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 70787.8us
I20250905 20:00:23.598306 30684 consensus_queue.cc:237] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:23.582316 31093 raft_consensus.cc:3058] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:23.594399 31200 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:23.604213 30148 raft_consensus.cc:2802] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election won for term 6
I20250905 20:00:23.579353 31084 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Advancing to term 10
I20250905 20:00:23.606194 30862 consensus_queue.cc:1035] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.006s
I20250905 20:00:23.613119 28051 leader_election.cc:304] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:23.606341 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:23.607434 30148 raft_consensus.cc:695] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
W20250905 20:00:23.607959 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 77530.1us
I20250905 20:00:23.603365 31195 raft_consensus.cc:3058] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 1 FOLLOWER]: Advancing to term 2
I20250905 20:00:23.609071 28158 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:36164
I20250905 20:00:23.620577 28158 raft_consensus.cc:604] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Received request to transfer leadership to TS cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:23.621260 28158 raft_consensus.cc:612] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250905 20:00:23.609865 28125 leader_election.cc:336] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.560s (SENT)
I20250905 20:00:23.624658 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:23.625319 28125 leader_election.cc:304] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:23.605783 31187 raft_consensus.cc:695] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:23.631462 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:23.636070 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:23.637182 28744 raft_consensus.cc:2466] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:23.640281 30148 consensus_queue.cc:237] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:23.641115 28123 leader_election.cc:304] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:23.591689 31177 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:23.642966 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:23.642328 30946 raft_consensus.cc:2747] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250905 20:00:23.645776 31083 raft_consensus.cc:3053] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Stepping down as leader of term 5
I20250905 20:00:23.615254 31200 raft_consensus.cc:3058] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:23.655987 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:23.581844 31193 consensus_queue.cc:1046] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.053s
W20250905 20:00:23.680768 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 73724.4us
I20250905 20:00:23.613718 31203 consensus_queue.cc:260] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:23.682411 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:23.655404 31087 raft_consensus.cc:2802] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:23.683861 26858 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20250905 20:00:23.684399 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 74586.8us
I20250905 20:00:23.659147 31083 raft_consensus.cc:738] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
W20250905 20:00:23.685715 28125 leader_election.cc:336] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.727s (SENT)
I20250905 20:00:23.685859 31083 consensus_queue.cc:260] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:23.686604 31210 raft_consensus.cc:2747] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:23.686951 31203 raft_consensus.cc:3058] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:23.687350 31083 raft_consensus.cc:3058] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:23.690806 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:23.691653 28744 raft_consensus.cc:3058] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:23.694832 30978 raft_consensus.cc:3053] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Stepping down as leader of term 5
I20250905 20:00:23.634632 31198 consensus_queue.cc:237] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:23.695380 30978 raft_consensus.cc:738] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:23.694926 26858 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250905 20:00:23.698544 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:23.699148 31087 raft_consensus.cc:491] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:23.699553 31087 raft_consensus.cc:3058] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:23.686220 28125 leader_election.cc:304] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:23.696563 30978 consensus_queue.cc:260] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:23.669176 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:23.614840 30775 consensus_queue.cc:260] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:23.735414 31175 raft_consensus.cc:513] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:23.738941 28744 raft_consensus.cc:2466] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
W20250905 20:00:23.739284 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:23.740381 28123 leader_election.cc:304] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:23.747103 31212 consensus_queue.cc:1035] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 20:00:23.751940 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:23.753001 28157 raft_consensus.cc:2408] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 8 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250905 20:00:23.753759 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:23.755065 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:23.756582 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:23.758272 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:23.759177 28151 raft_consensus.cc:2391] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 5: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:23.760362 28156 raft_consensus.cc:1238] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:23.761608 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:23.762523 28154 raft_consensus.cc:2466] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 6.
I20250905 20:00:23.764406 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:23.765331 28159 raft_consensus.cc:3058] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
W20250905 20:00:23.771107 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 158047us
I20250905 20:00:23.731218 31193 raft_consensus.cc:2747] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:23.702754 31178 leader_election.cc:290] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:23.712124 31208 raft_consensus.cc:3058] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Advancing to term 7
W20250905 20:00:23.775213 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:23.776360 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 20:00:23.778621 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 92951.4us
I20250905 20:00:23.738291 28746 raft_consensus.cc:2466] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 8.
I20250905 20:00:23.793258 28051 leader_election.cc:304] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:23.798695 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:23.776787 30946 raft_consensus.cc:2802] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader election won for term 7
I20250905 20:00:23.800060 28007 raft_consensus.cc:2466] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 20:00:23.802309 28159 raft_consensus.cc:2466] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 6.
I20250905 20:00:23.785769 30978 raft_consensus.cc:3058] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:23.778846 31197 raft_consensus.cc:695] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 5 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
W20250905 20:00:23.806919 28125 leader_election.cc:336] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.882s (SENT)
I20250905 20:00:23.791093 31213 raft_consensus.cc:2802] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 20:00:23.812654 31213 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:23.813086 31213 raft_consensus.cc:3058] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:23.816663 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 20:00:23.805780 31196 raft_consensus.cc:2762] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
W20250905 20:00:23.810345 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:23.820021 28123 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:23.821154 31178 raft_consensus.cc:2802] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250905 20:00:23.821552 31178 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:23.821887 31178 raft_consensus.cc:3058] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Advancing to term 9
W20250905 20:00:23.827142 31199 log.cc:927] Time spent T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.356s user 0.003s sys 0.000s
W20250905 20:00:23.827795 31189 consensus_meta.cc:287] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.513s user 0.006s sys 0.006s
I20250905 20:00:23.828222 31189 raft_consensus.cc:513] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:23.783102 31187 consensus_queue.cc:237] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:23.784881 31176 consensus_queue.cc:1035] T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20250905 20:00:23.825666 31195 raft_consensus.cc:513] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:23.833137 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:23.777962 31220 raft_consensus.cc:491] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
W20250905 20:00:23.834784 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:23.841281 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 20:00:23.842316 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:23.809669 30946 raft_consensus.cc:695] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:23.857295 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:23.858314 28007 raft_consensus.cc:2374] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5. Re-sending same reply.
W20250905 20:00:23.861629 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 58693.2us
I20250905 20:00:23.864557 28153 raft_consensus.cc:1238] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
I20250905 20:00:23.833211 31175 leader_election.cc:290] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:23.870009 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 92084.8us
I20250905 20:00:23.872144 28051 leader_election.cc:304] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
W20250905 20:00:23.882311 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:23.889014 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:23.890007 28007 raft_consensus.cc:2466] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
I20250905 20:00:23.894372 28051 leader_election.cc:304] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:23.890161 30775 raft_consensus.cc:3058] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:23.890877 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:23.904626 28151 raft_consensus.cc:2391] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 4: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:23.908948 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:23.867743 31177 leader_election.cc:290] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:23.916662 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 383901us
I20250905 20:00:23.897079 31188 raft_consensus.cc:2802] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 20:00:23.899348 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:23.899983 31189 leader_election.cc:290] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:23.923780 28050 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:23.926653 28050 leader_election.cc:304] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 20:00:23.927171 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:23.930097 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:23.929975 28747 raft_consensus.cc:1238] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:23.932703 28717 leader_election.cc:304] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:23.934983 28051 leader_election.cc:304] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 20:00:23.931483 28125 leader_election.cc:336] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 2.439s (SENT)
I20250905 20:00:23.935791 28125 leader_election.cc:304] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:23.932072 28050 leader_election.cc:304] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:23.941968 28005 raft_consensus.cc:2408] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20250905 20:00:23.943395 31099 raft_consensus.cc:3053] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 LEADER]: Stepping down as leader of term 7
I20250905 20:00:23.943867 31099 raft_consensus.cc:738] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:23.940464 31203 raft_consensus.cc:2802] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250905 20:00:23.944789 31099 consensus_queue.cc:260] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:23.945611 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:23.940680 30777 raft_consensus.cc:2747] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:23.947219 30777 raft_consensus.cc:2802] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election won for term 6
I20250905 20:00:23.947736 30777 raft_consensus.cc:695] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:23.948690 30777 consensus_queue.cc:237] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:23.938092 31177 raft_consensus.cc:3058] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:23.951884 31216 raft_consensus.cc:2802] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:23.952536 31216 raft_consensus.cc:491] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:23.922122 31197 consensus_queue.cc:237] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:23.952915 31216 raft_consensus.cc:3058] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:23.955212 28009 raft_consensus.cc:1238] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 LEADER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:23.930629 31196 raft_consensus.cc:2802] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 20:00:23.868919 31220 raft_consensus.cc:513] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:23.953012 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:23.921674 31188 raft_consensus.cc:491] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:23.957378 28744 raft_consensus.cc:2391] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 8: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:23.929551 31217 raft_consensus.cc:2747] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250905 20:00:23.976372 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:23.978015 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:23.946269 31099 raft_consensus.cc:3058] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:23.949810 31200 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:23.991890 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 65753.2us
I20250905 20:00:24.093658 28005 raft_consensus.cc:1238] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:24.094230 31200 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:23.944836 31203 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:23.962862 31196 raft_consensus.cc:491] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:24.077119 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:24.078329 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:24.096367 28744 raft_consensus.cc:3058] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:23.895092 30946 consensus_queue.cc:237] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:23.967752 28746 raft_consensus.cc:2391] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 7: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 20:00:23.933619 31175 consensus_queue.cc:1046] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.050s
I20250905 20:00:23.957060 31189 consensus_queue.cc:1046] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.128s
W20250905 20:00:24.084792 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:23.983084 31188 raft_consensus.cc:3058] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
W20250905 20:00:24.088608 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 156582us
I20250905 20:00:23.979357 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:24.102766 28717 leader_election.cc:304] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:24.103193 28002 raft_consensus.cc:2466] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 20:00:24.104210 31196 raft_consensus.cc:3058] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:24.114097 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:24.115564 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:24.116889 28002 raft_consensus.cc:3058] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Advancing to term 10
I20250905 20:00:24.117515 28747 raft_consensus.cc:1273] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20250905 20:00:24.118196 28051 connection.cc:232] RPC call timeout handler was delayed by -2.8262589400000024s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20230000000000004s for a response
W20250905 20:00:24.118736 28051 connection.cc:232] RPC call timeout handler was delayed by -2.0053447349999942s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1514s for a response
W20250905 20:00:24.119107 28051 connection.cc:232] RPC call timeout handler was delayed by -1.4743447350000451s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21040000000000003s for a response
W20250905 20:00:24.119390 28051 connection.cc:232] RPC call timeout handler was delayed by -1.4149447350000628s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.217s for a response
W20250905 20:00:24.119668 28051 connection.cc:232] RPC call timeout handler was delayed by -1.3888447350000206s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21989999999999998s for a response
W20250905 20:00:24.119952 28051 connection.cc:232] RPC call timeout handler was delayed by -1.1683447350000051s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2444s for a response
I20250905 20:00:24.121726 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:24.121300 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:24.122406 28076 raft_consensus.cc:2466] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 8.
I20250905 20:00:24.122606 28085 raft_consensus.cc:2466] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 7.
I20250905 20:00:24.123186 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:24.125490 28084 raft_consensus.cc:1238] T aace14abd237420c8211ce611ea09f0e P e4413ae5e0b64b52a6f360ac580239b9 [term 7 LEADER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:24.108080 31195 leader_election.cc:290] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:24.128870 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:24.104075 31203 raft_consensus.cc:3058] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Advancing to term 9
I20250905 20:00:24.133299 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:24.133967 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:24.135597 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:24.136138 28077 raft_consensus.cc:2466] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 20:00:24.136819 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:24.121272 31192 log.cc:927] Time spent T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410: Append to log took a long time: real 0.110s user 0.000s sys 0.013s
I20250905 20:00:24.139994 28076 raft_consensus.cc:1273] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20250905 20:00:24.144001 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:24.144694 28085 raft_consensus.cc:2374] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6. Re-sending same reply.
I20250905 20:00:24.145988 31220 leader_election.cc:290] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:24.147745 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:24.148190 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:24.148430 28083 raft_consensus.cc:2374] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 11. Re-sending same reply.
W20250905 20:00:24.149348 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:24.149755 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:24.127279 31200 raft_consensus.cc:491] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:24.153106 28082 raft_consensus.cc:1273] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250905 20:00:24.161417 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:24.163364 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:24.163960 28077 raft_consensus.cc:2374] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5. Re-sending same reply.
I20250905 20:00:24.166481 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:24.167464 28159 raft_consensus.cc:2391] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 7: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:24.169078 28153 raft_consensus.cc:1238] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20250905 20:00:24.130101 31083 consensus_queue.cc:1035] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.033s
I20250905 20:00:24.178975 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:24.129783 31190 consensus_queue.cc:1035] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20250905 20:00:24.239815 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:24.187753 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:24.200349 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:24.379731 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:24.218281 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:24.195616 28718 leader_election.cc:336] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 2.846s (SENT)
I20250905 20:00:24.179425 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:24.366926 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:24.242756 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:24.385457 28006 raft_consensus.cc:2391] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 5: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
W20250905 20:00:24.292759 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:24.322934 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:24.222318 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:24.388265 28005 raft_consensus.cc:2366] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a for earlier term 6. Current term is 7.
I20250905 20:00:24.271679 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:24.387153 28744 raft_consensus.cc:2466] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 7.
I20250905 20:00:24.396720 28153 raft_consensus.cc:2391] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 8: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:24.387387 28154 raft_consensus.cc:2391] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 2: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:24.179888 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:24.218063 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 70523.3us
W20250905 20:00:24.240615 31225 log.cc:927] Time spent T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.152s user 0.000s sys 0.001s
I20250905 20:00:24.388737 28159 raft_consensus.cc:2391] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 5: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:24.321311 28749 raft_consensus.cc:2466] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 4.
W20250905 20:00:24.383930 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:24.395148 28158 raft_consensus.cc:2391] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 8: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:24.355787 28002 raft_consensus.cc:2466] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 10.
I20250905 20:00:24.166412 31171 raft_consensus.cc:491] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:24.401700 28155 raft_consensus.cc:1238] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 6. Current term is 7. Ops: [6.7-6.7]
W20250905 20:00:24.217077 31230 log.cc:927] Time spent T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.133s user 0.001s sys 0.000s
W20250905 20:00:24.409452 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:24.410766 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:24.411207 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:24.411763 28157 raft_consensus.cc:2391] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 8: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:24.413458 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:24.414295 28083 raft_consensus.cc:3058] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:24.166937 31200 raft_consensus.cc:513] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:24.242285 31189 consensus_queue.cc:1046] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.174s
I20250905 20:00:24.415518 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:24.118518 31224 consensus_queue.cc:1035] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.442s
W20250905 20:00:24.465603 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:24.420190 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:24.456318 28008 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 11.
I20250905 20:00:24.419979 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:24.478106 28154 raft_consensus.cc:2391] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 6: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
W20250905 20:00:24.444361 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:24.507848 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:24.508342 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:24.520668 28084 raft_consensus.cc:2466] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 9.
W20250905 20:00:24.410115 31213 consensus_meta.cc:287] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.597s user 0.013s sys 0.000s
W20250905 20:00:24.435319 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 239758us
I20250905 20:00:24.417515 31238 consensus_queue.cc:1046] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20250905 20:00:24.419545 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:24.480660 27975 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.881s user 0.015s sys 0.028s
I20250905 20:00:24.481837 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:24.522455 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:24.530225 27975 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.930s user 0.015s sys 0.028s
I20250905 20:00:24.530668 28002 raft_consensus.cc:2391] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 5: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:24.522949 28049 leader_election.cc:304] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
W20250905 20:00:24.531118 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 949735us
I20250905 20:00:24.506738 28003 raft_consensus.cc:2466] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 7.
I20250905 20:00:24.409986 31210 raft_consensus.cc:2802] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250905 20:00:24.526762 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:24.530485 28006 raft_consensus.cc:2466] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 20:00:24.438017 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:24.527760 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:24.524296 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 212714us
I20250905 20:00:24.547691 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:24.548719 28084 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
I20250905 20:00:24.311146 31178 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:24.553184 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:24.547740 28008 raft_consensus.cc:2374] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 11 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 11. Re-sending same reply.
I20250905 20:00:24.536684 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:24.441979 31174 consensus_meta.cc:287] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 1.041s user 0.005s sys 0.006s
W20250905 20:00:24.420519 31093 consensus_meta.cc:287] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.794s user 0.011s sys 0.000s
W20250905 20:00:24.216507 27894 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet fc7dc431504743d79dbdc8eea83f9dcb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 26b2f874b8e54f08b01a9ed00d5cce05 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 20:00:24.416458 31171 raft_consensus.cc:513] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:24.546851 28082 raft_consensus.cc:2466] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 20:00:24.560540 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 20:00:25.008392 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 485553us
W20250905 20:00:25.033759 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 464451us
W20250905 20:00:25.077093 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:24.974086 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:25.133402 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:25.134527 28153 raft_consensus.cc:2466] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:24.619354 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:24.621547 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:25.136121 28155 raft_consensus.cc:2408] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20250905 20:00:25.136175 28001 raft_consensus.cc:2366] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a for earlier term 6. Current term is 7.
W20250905 20:00:24.814779 28718 leader_election.cc:336] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.736s (SENT)
I20250905 20:00:24.983011 30777 raft_consensus.cc:2802] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader election won for term 4
W20250905 20:00:25.137331 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 322574us
I20250905 20:00:25.137995 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:25.144690 28001 raft_consensus.cc:2374] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 11. Re-sending same reply.
W20250905 20:00:25.145355 28718 leader_election.cc:336] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.742s (SENT)
I20250905 20:00:25.146191 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:25.146881 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:25.139359 28081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 346) took 1017 ms. Trace:
I20250905 20:00:25.147293 28001 raft_consensus.cc:2466] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 7.
I20250905 20:00:24.711227 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 20:00:24.562973 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:24.688794 28083 raft_consensus.cc:2466] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:25.151383 28003 raft_consensus.cc:2374] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 10. Re-sending same reply.
W20250905 20:00:25.151816 27975 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:24.692051 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:24.691579 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:24.623796 30946 consensus_queue.cc:1035] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.553s
W20250905 20:00:24.906793 31226 log.cc:927] Time spent T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.177s user 0.000s sys 0.001s
I20250905 20:00:25.155372 28154 raft_consensus.cc:2391] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 8: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:25.155982 31171 leader_election.cc:290] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:25.142302 28084 raft_consensus.cc:1238] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 9. Current term is 10. Ops: []
I20250905 20:00:24.554286 31213 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:24.567519 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:24.661895 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:25.158865 31193 raft_consensus.cc:491] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
W20250905 20:00:25.154217 28749 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.549s user 0.007s sys 0.009s
I20250905 20:00:24.620131 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:24.550963 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 49550.4us
I20250905 20:00:24.619640 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:24.814129 30978 consensus_meta.cc:287] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.983s user 0.013s sys 0.000s
W20250905 20:00:25.138610 27894 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 1.02609e+06us
W20250905 20:00:25.008674 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 478940us
W20250905 20:00:24.997699 30775 consensus_meta.cc:287] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 1.065s user 0.016s sys 0.002s
I20250905 20:00:25.147244 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:25.148252 28718 leader_election.cc:336] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.911s (SENT)
W20250905 20:00:25.168045 28048 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.583s user 0.012s sys 0.000s
W20250905 20:00:25.203498 28048 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.642s user 0.012s sys 0.000s
I20250905 20:00:25.203449 28718 leader_election.cc:304] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:24.586020 27974 leader_election.cc:304] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:25.148010 28081 rpcz_store.cc:276] 0905 20:00:24.121531 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:24.129504 (+ 7973us) service_pool.cc:225] Handling call
0905 20:00:24.129854 (+ 350us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:24.142585 (+ 12731us) raft_consensus.cc:1532] Early marking committed up to index 7
0905 20:00:24.389879 (+247294us) log.cc:844] Serialized 14 byte log entry
0905 20:00:25.138881 (+749002us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:25.138947 (+ 66us) raft_consensus.cc:1643] Marking committed up to 7
0905 20:00:25.139022 (+ 75us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:25.139114 (+ 92us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:25.139282 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":58793,"prepare.queue_time_us":326816,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":49,"raft.queue_time_us":355108,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":117,"spinlock_wait_cycles":3840,"thread_start_us":636990,"threads_started":3}
I20250905 20:00:24.619891 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:24.716682 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:25.160368 28749 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.555s user 0.007s sys 0.009s
I20250905 20:00:25.164268 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:25.165592 30777 raft_consensus.cc:695] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 4 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:25.169554 28006 raft_consensus.cc:1273] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20250905 20:00:25.251212 28154 raft_consensus.cc:2391] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 6: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
W20250905 20:00:25.268822 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 71455.1us
W20250905 20:00:25.171125 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:25.075922 31087 consensus_meta.cc:287] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.376s user 0.010s sys 0.003s
W20250905 20:00:24.880038 31222 log.cc:927] Time spent T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.162s user 0.000s sys 0.000s
W20250905 20:00:24.998073 31208 consensus_meta.cc:287] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 1.207s user 0.018s sys 0.000s
I20250905 20:00:25.272552 28008 raft_consensus.cc:1273] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250905 20:00:25.272814 28155 raft_consensus.cc:2391] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 5: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
W20250905 20:00:24.924094 31177 consensus_meta.cc:287] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.956s user 0.008s sys 0.002s
W20250905 20:00:25.104339 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:25.298707 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:25.301757 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:25.302678 28002 raft_consensus.cc:2466] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 7.
W20250905 20:00:25.307044 28715 leader_election.cc:336] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.736s (SENT)
I20250905 20:00:25.307225 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:25.307812 28715 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:25.308498 28159 raft_consensus.cc:2408] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20250905 20:00:25.309257 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:25.310017 28157 raft_consensus.cc:2374] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 7. Re-sending same reply.
I20250905 20:00:25.311333 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:25.275362 28156 raft_consensus.cc:2391] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 9: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:25.276902 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:25.296236 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 147991us
I20250905 20:00:25.020905 31238 raft_consensus.cc:491] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:25.009086 31221 raft_consensus.cc:3053] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 LEADER]: Stepping down as leader of term 4
I20250905 20:00:25.327474 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:25.299892 28080 raft_consensus.cc:2466] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
W20250905 20:00:25.314673 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 155865us
I20250905 20:00:25.206774 31190 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:25.159499 31193 raft_consensus.cc:513] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:25.337734 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:25.361967 31151 raft_consensus.cc:491] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:25.389982 31245 log.cc:927] Time spent T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.370s user 0.001s sys 0.000s
I20250905 20:00:25.388708 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:25.389207 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:25.393149 28749 raft_consensus.cc:1273] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250905 20:00:25.160315 31093 raft_consensus.cc:513] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:25.357450 28155 pending_rounds.cc:85] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting all ops after (but not including) 4
I20250905 20:00:25.429438 28155 pending_rounds.cc:107] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20250905 20:00:25.429731 28155 raft_consensus.cc:2887] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250905 20:00:25.389071 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:25.363518 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:25.314139 31099 consensus_meta.cc:287] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.282s user 0.006s sys 0.007s
I20250905 20:00:25.354099 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:25.288075 31171 raft_consensus.cc:491] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:25.390940 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:25.168895 31174 raft_consensus.cc:513] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:25.393992 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:25.274714 31083 raft_consensus.cc:491] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:25.340150 28718 leader_election.cc:336] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 2.006s (SENT)
I20250905 20:00:25.430976 28006 raft_consensus.cc:2408] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250905 20:00:25.356401 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:25.297279 31208 raft_consensus.cc:2747] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:25.291813 31087 raft_consensus.cc:513] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:25.314352 31177 raft_consensus.cc:2747] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250905 20:00:25.434197 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 135167us
I20250905 20:00:25.560432 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:25.429330 31200 leader_election.cc:290] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:25.565635 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:25.566223 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:25.567215 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:25.567584 28084 raft_consensus.cc:2408] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20250905 20:00:25.567829 28076 raft_consensus.cc:2374] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6. Re-sending same reply.
I20250905 20:00:25.438481 28004 raft_consensus.cc:2391] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 7: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
W20250905 20:00:25.362332 31237 log.cc:927] Time spent T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.356s user 0.000s sys 0.001s
I20250905 20:00:25.568883 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:25.329370 31221 raft_consensus.cc:738] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:25.569576 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:25.570258 28004 raft_consensus.cc:2466] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 6.
I20250905 20:00:25.570535 31221 consensus_queue.cc:260] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:25.571259 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:25.571648 31221 raft_consensus.cc:3058] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:25.571872 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:25.572654 28083 raft_consensus.cc:2391] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 5: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 20:00:25.329229 30978 raft_consensus.cc:2802] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader election won for term 7
I20250905 20:00:25.573230 30978 raft_consensus.cc:695] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:25.573582 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:25.574056 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:25.575199 28083 raft_consensus.cc:1273] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250905 20:00:25.570546 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:25.441588 28152 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.013s sys 0.007s
I20250905 20:00:25.453671 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:25.445034 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 144028us
I20250905 20:00:25.444545 28151 raft_consensus.cc:2466] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 8.
I20250905 20:00:25.390194 31190 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:25.361583 31234 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.606s user 0.046s sys 0.045s
W20250905 20:00:25.500511 27975 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:25.549669 28158 raft_consensus.cc:1811] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Handling vote request from an unknown peer cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 20:00:25.495803 28122 leader_election.cc:336] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.503s (SENT)
I20250905 20:00:25.582773 28153 raft_consensus.cc:3058] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Advancing to term 8
W20250905 20:00:25.582763 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 134440us
I20250905 20:00:25.497416 28008 raft_consensus.cc:2391] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 10: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:25.436149 31151 raft_consensus.cc:513] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:25.390741 31256 raft_consensus.cc:491] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:25.590399 31256 raft_consensus.cc:513] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:25.552047 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 190107us
W20250905 20:00:25.594563 31232 log.cc:927] Time spent T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.975s user 0.000s sys 0.001s
I20250905 20:00:25.596499 31151 leader_election.cc:290] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:25.291458 31224 raft_consensus.cc:2747] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:25.597229 31151 raft_consensus.cc:2747] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20250905 20:00:25.559422 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 973604us
I20250905 20:00:25.570955 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:25.572463 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 20:00:25.407014 31189 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.714s user 0.020s sys 0.040s
W20250905 20:00:25.301035 31203 consensus_meta.cc:287] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 1.154s user 0.012s sys 0.000s
I20250905 20:00:25.574198 30978 consensus_queue.cc:237] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:25.443274 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:25.441992 28154 raft_consensus.cc:1238] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:25.575112 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 20:00:25.576251 28152 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.658s user 0.013s sys 0.007s
I20250905 20:00:25.580643 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 20:00:25.550630 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 210527us
W20250905 20:00:25.605379 28718 leader_election.cc:336] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 2.019s (SENT)
I20250905 20:00:25.605901 28718 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 20:00:25.583644 28122 leader_election.cc:336] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.531s (SENT)
I20250905 20:00:25.584394 28158 raft_consensus.cc:2408] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20250905 20:00:25.584532 28077 raft_consensus.cc:2466] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
I20250905 20:00:25.456771 31099 raft_consensus.cc:491] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:25.612164 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:25.586766 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:25.587792 27975 leader_election.cc:304] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:25.596657 28007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 373) took 1676 ms. Trace:
I20250905 20:00:25.612950 28008 raft_consensus.cc:2466] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 20:00:25.359030 31238 raft_consensus.cc:513] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:25.594652 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:25.600754 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:25.602056 28750 raft_consensus.cc:3053] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 5 LEADER]: Stepping down as leader of term 5
I20250905 20:00:25.627620 28750 raft_consensus.cc:738] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:25.606933 28122 leader_election.cc:304] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:25.608628 28003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 395) took 1129 ms. Trace:
I20250905 20:00:25.566617 28006 raft_consensus.cc:2391] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 5: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:25.604117 28082 raft_consensus.cc:2374] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 10. Re-sending same reply.
I20250905 20:00:25.577169 31195 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:25.582178 31234 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.853s user 0.054s sys 0.053s
I20250905 20:00:25.585819 28010 raft_consensus.cc:2391] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 8: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:25.612187 28158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33752 (request call id 1726) took 1040 ms. Trace:
I20250905 20:00:25.637072 28750 consensus_queue.cc:260] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:25.613237 28007 rpcz_store.cc:276] 0905 20:00:23.922681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:23.925520 (+ 2839us) service_pool.cc:225] Handling call
0905 20:00:23.925934 (+ 414us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:23.956368 (+ 30434us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:23.956404 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:23.963121 (+ 6717us) log.cc:844] Serialized 26 byte log entry
0905 20:00:24.129961 (+166840us) raft_consensus.cc:1643] Marking committed up to 9
0905 20:00:24.385586 (+255625us) log.cc:844] Serialized 14 byte log entry
0905 20:00:24.387735 (+ 2149us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:24.387825 (+ 90us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:25.596411 (+1208586us) raft_consensus.cc:1683] finished
0905 20:00:25.596431 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:25.596566 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":393068,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":26132,"raft.queue_time_us":16883,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":7816,"thread_start_us":164324,"threads_started":2,"wal-append.queue_time_us":232500}
W20250905 20:00:25.631044 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:25.651427 28718 leader_election.cc:336] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 2.206s (SENT)
I20250905 20:00:25.650156 28158 rpcz_store.cc:276] 0905 20:00:24.572171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:25.294130 (+721959us) service_pool.cc:225] Handling call
0905 20:00:25.612122 (+317992us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 20:00:25.651445 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:25.653450 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:25.628880 28003 rpcz_store.cc:276] 0905 20:00:24.479892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:25.152266 (+672374us) service_pool.cc:225] Handling call
0905 20:00:25.152629 (+ 363us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:25.153034 (+ 405us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:25.153061 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:25.153547 (+ 486us) log.cc:844] Serialized 26 byte log entry
0905 20:00:25.392391 (+238844us) raft_consensus.cc:1643] Marking committed up to 6
0905 20:00:25.607855 (+215464us) log.cc:844] Serialized 14 byte log entry
0905 20:00:25.608188 (+ 333us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:25.608295 (+ 107us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:25.608357 (+ 62us) raft_consensus.cc:1683] finished
0905 20:00:25.608375 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:25.608549 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":241,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":81,"thread_start_us":406052,"threads_started":2,"wal-append.queue_time_us":272243}
W20250905 20:00:25.653875 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:25.654551 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:25.655048 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:25.559754 31171 raft_consensus.cc:513] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:25.657037 28004 raft_consensus.cc:1238] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250905 20:00:25.663942 28750 raft_consensus.cc:3058] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:25.559921 31083 raft_consensus.cc:513] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:25.657895 28718 leader_election.cc:336] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 2.240s (SENT)
W20250905 20:00:25.664254 28125 leader_election.cc:336] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.575s (SENT)
W20250905 20:00:25.611754 31189 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.980s user 0.020s sys 0.041s
I20250905 20:00:25.659965 27975 leader_election.cc:304] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:25.669286 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:25.684266 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:25.689770 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:25.632681 31084 raft_consensus.cc:2747] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:25.734967 28150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2291) took 1346 ms. Trace:
I20250905 20:00:25.735430 28150 rpcz_store.cc:276] 0905 20:00:24.388916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:24.390105 (+ 1189us) service_pool.cc:225] Handling call
0905 20:00:24.390620 (+ 515us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:24.390987 (+ 367us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:24.391026 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:24.399228 (+ 8202us) log.cc:844] Serialized 26 byte log entry
0905 20:00:24.528691 (+129463us) raft_consensus.cc:1643] Marking committed up to 7
0905 20:00:24.528738 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:24.528814 (+ 76us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:25.734714 (+1205900us) raft_consensus.cc:1683] finished
0905 20:00:25.734738 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:25.734904 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":100103,"threads_started":1,"wal-append.queue_time_us":310602}
I20250905 20:00:25.656131 31151 raft_consensus.cc:2747] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250905 20:00:25.738633 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:25.740331 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:25.741343 28002 raft_consensus.cc:2466] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 7.
I20250905 20:00:25.741799 28154 raft_consensus.cc:2466] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
I20250905 20:00:25.741902 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 20:00:25.742874 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 78659.4us
I20250905 20:00:25.578243 30777 consensus_queue.cc:237] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:25.758730 28125 leader_election.cc:336] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.658s (SENT)
I20250905 20:00:25.754072 28750 raft_consensus.cc:1273] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250905 20:00:25.631654 31099 raft_consensus.cc:513] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:25.625830 31203 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:25.672078 31267 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:25.757115 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:25.774294 28125 leader_election.cc:336] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.717s (SENT)
I20250905 20:00:25.733067 31220 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:25.783713 28154 raft_consensus.cc:1273] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250905 20:00:25.781599 28158 raft_consensus.cc:1273] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20250905 20:00:25.770215 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:25.738739 31177 raft_consensus.cc:491] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:25.759315 31196 consensus_meta.cc:287] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 1.632s user 0.013s sys 0.001s
I20250905 20:00:25.666572 31195 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:25.789896 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 145320us
W20250905 20:00:25.767401 31259 log.cc:927] Time spent T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.085s user 0.000s sys 0.001s
I20250905 20:00:25.794992 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:25.797130 28082 raft_consensus.cc:2391] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 5: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:25.801021 28744 raft_consensus.cc:1238] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 8. Current term is 9. Ops: []
W20250905 20:00:25.802865 28125 leader_election.cc:336] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.833s (SENT)
W20250905 20:00:25.804015 31242 log.cc:927] Time spent T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.250s user 0.000s sys 0.001s
I20250905 20:00:25.806360 28152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39586 (request call id 407) took 1643 ms. Trace:
I20250905 20:00:25.806492 31256 leader_election.cc:290] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:25.806766 31174 leader_election.cc:290] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:25.799047 31196 raft_consensus.cc:513] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:25.807466 31254 log.cc:927] Time spent T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.142s user 0.000s sys 0.000s
I20250905 20:00:25.811614 31083 leader_election.cc:290] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:25.813329 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:25.814337 28743 raft_consensus.cc:1273] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250905 20:00:25.768257 31261 raft_consensus.cc:2747] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250905 20:00:25.815403 28159 raft_consensus.cc:1273] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250905 20:00:25.815702 28001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 394) took 1393 ms. Trace:
I20250905 20:00:25.767808 31084 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:25.768160 31178 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:25.821583 28152 rpcz_store.cc:276] 0905 20:00:24.162735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:24.163067 (+ 332us) service_pool.cc:225] Handling call
0905 20:00:24.379664 (+216597us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:24.393183 (+ 13519us) raft_consensus.cc:1532] Early marking committed up to index 7
0905 20:00:24.717905 (+324722us) log.cc:844] Serialized 14 byte log entry
0905 20:00:25.605023 (+887118us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:25.622919 (+ 17896us) log.cc:844] Serialized 26 byte log entry
0905 20:00:25.624412 (+ 1493us) raft_consensus.cc:1643] Marking committed up to 7
0905 20:00:25.624486 (+ 74us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:25.624589 (+ 103us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:25.806094 (+181505us) raft_consensus.cc:1683] finished
0905 20:00:25.806122 (+ 28us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:25.806289 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":756794,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":57823,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":13255,"thread_start_us":988940,"threads_started":2,"wal-append.queue_time_us":806861}
W20250905 20:00:25.864710 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 281053us
W20250905 20:00:25.893365 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 74140.2us
I20250905 20:00:25.981729 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 20:00:25.943929 28715 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.000s sys 0.021s
I20250905 20:00:25.982561 28157 raft_consensus.cc:2408] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
W20250905 20:00:25.951381 28717 leader_election.cc:336] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Timed out: RequestConsensusVote RPC to 127.26.58.131:36705 timed out after 1.976s (SENT)
W20250905 20:00:25.979580 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 188860us
I20250905 20:00:25.826472 28085 raft_consensus.cc:1238] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 LEADER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 3. Current term is 4. Ops: []
W20250905 20:00:25.984475 28125 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.860s (SENT) [suppressed 84 similar messages]
I20250905 20:00:25.830367 28158 raft_consensus.cc:1238] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 4. Current term is 5. Ops: []
W20250905 20:00:25.985778 28717 leader_election.cc:336] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705): Timed out: RequestConsensusVote RPC to 127.26.58.131:36705 timed out after 2.006s (SENT)
I20250905 20:00:25.986406 28717 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:25.979979 28001 rpcz_store.cc:276] 0905 20:00:24.422374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:25.150968 (+728594us) service_pool.cc:225] Handling call
0905 20:00:25.153409 (+ 2441us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:25.153751 (+ 342us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:25.161408 (+ 7657us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:25.277888 (+116480us) log.cc:844] Serialized 26 byte log entry
0905 20:00:25.564564 (+286676us) raft_consensus.cc:1643] Marking committed up to 7
0905 20:00:25.781940 (+217376us) log.cc:844] Serialized 14 byte log entry
0905 20:00:25.782200 (+ 260us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:25.782299 (+ 99us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:25.815416 (+ 33117us) raft_consensus.cc:1683] finished
0905 20:00:25.815442 (+ 26us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:25.815599 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":237614,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":67,"raft.queue_time_us":182092,"raft.run_cpu_time_us":73,"raft.run_wall_time_us":74,"thread_start_us":366412,"threads_started":2,"wal-append.queue_time_us":303934}
W20250905 20:00:25.979339 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 185358us
I20250905 20:00:25.989617 27975 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:25.835448 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:25.982947 28715 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.599s user 0.001s sys 0.021s
I20250905 20:00:25.993819 28155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1731) took 1183 ms. Trace:
I20250905 20:00:25.995442 28153 raft_consensus.cc:2466] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 8.
I20250905 20:00:25.995550 28155 rpcz_store.cc:276] 0905 20:00:24.799002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:25.300512 (+501510us) service_pool.cc:225] Handling call
0905 20:00:25.357207 (+ 56695us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:25.452543 (+ 95336us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:25.452579 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:25.453219 (+ 640us) log.cc:844] Serialized 26 byte log entry
0905 20:00:25.662047 (+208828us) raft_consensus.cc:1643] Marking committed up to 4
0905 20:00:25.662093 (+ 46us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:25.662188 (+ 95us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:25.981652 (+319464us) raft_consensus.cc:1683] finished
0905 20:00:25.981690 (+ 38us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:25.981903 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":67089,"threads_started":1,"wal-append.queue_time_us":310856}
W20250905 20:00:25.985308 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:25.981058 28122 leader_election.cc:336] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.546s (SENT)
I20250905 20:00:26.005210 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:26.018306 28085 raft_consensus.cc:2391] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 8: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
W20250905 20:00:26.018370 28718 leader_election.cc:336] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 2.178s (SENT)
I20250905 20:00:26.018941 28718 leader_election.cc:304] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:26.020129 27975 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
W20250905 20:00:25.912981 28716 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:26.020865 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:26.021648 28750 raft_consensus.cc:3058] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Advancing to term 7
W20250905 20:00:26.022528 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:26.026820 27975 leader_election.cc:304] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:25.797101 31177 raft_consensus.cc:513] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:26.025928 31234 raft_consensus.cc:2747] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250905 20:00:26.038353 31174 consensus_queue.cc:1046] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
W20250905 20:00:26.043532 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 736455us
I20250905 20:00:25.852782 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
W20250905 20:00:26.046144 31188 consensus_meta.cc:287] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 1.933s user 0.010s sys 0.005s
I20250905 20:00:26.030787 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:26.046676 31188 raft_consensus.cc:513] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:26.047410 28747 raft_consensus.cc:2466] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 8.
W20250905 20:00:26.029567 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 48446.3us
I20250905 20:00:26.048761 31188 leader_election.cc:290] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:26.049301 28122 leader_election.cc:336] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.554s (SENT)
W20250905 20:00:26.049926 28122 leader_election.cc:336] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.658s (SENT)
I20250905 20:00:26.050272 28122 leader_election.cc:304] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:26.054911 28082 raft_consensus.cc:2391] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 5: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:25.796618 31220 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:26.057847 31220 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:25.886344 31276 raft_consensus.cc:2762] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20250905 20:00:26.058984 31276 raft_consensus.cc:3058] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:26.017235 31083 raft_consensus.cc:2747] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250905 20:00:26.064111 31216 consensus_meta.cc:287] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 2.111s user 0.016s sys 0.000s
I20250905 20:00:26.064687 31083 raft_consensus.cc:2802] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:26.065426 31083 raft_consensus.cc:491] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:26.065832 31083 raft_consensus.cc:3058] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:26.068475 28078 raft_consensus.cc:1238] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20250905 20:00:26.069550 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:26.069993 28157 raft_consensus.cc:1273] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250905 20:00:26.070453 28079 raft_consensus.cc:2466] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
I20250905 20:00:25.963294 31099 leader_election.cc:290] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:26.036907 28159 raft_consensus.cc:1238] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:25.959583 31203 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 9 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:25.917258 31093 leader_election.cc:290] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:25.964684 31266 raft_consensus.cc:491] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:26.033142 28154 raft_consensus.cc:1238] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 LEADER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:26.077910 28750 raft_consensus.cc:2466] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
I20250905 20:00:25.887559 31171 leader_election.cc:290] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:25.886447 31087 leader_election.cc:290] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:25.944522 31238 leader_election.cc:290] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:26.080255 28080 raft_consensus.cc:3053] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 LEADER]: Stepping down as leader of term 6
W20250905 20:00:26.081014 28051 leader_election.cc:336] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.514s (SENT)
I20250905 20:00:26.100530 28080 raft_consensus.cc:738] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:26.101163 28051 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:26.081240 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:25.951040 31190 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:26.084610 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:25.886476 31193 leader_election.cc:290] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:26.102186 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:26.089332 31234 consensus_queue.cc:1046] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20250905 20:00:26.090885 28122 leader_election.cc:336] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.717s (SENT)
I20250905 20:00:26.018921 31263 raft_consensus.cc:491] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:25.790680 31267 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:26.037503 31151 raft_consensus.cc:2747] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250905 20:00:26.088668 28718 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:26.043129 31177 leader_election.cc:290] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:26.106185 31221 consensus_meta.cc:287] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.534s user 0.013s sys 0.000s
I20250905 20:00:26.102293 28085 raft_consensus.cc:2466] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
I20250905 20:00:26.109747 28155 raft_consensus.cc:2391] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 7: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
W20250905 20:00:26.112684 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 79217.2us
I20250905 20:00:26.111583 28080 consensus_queue.cc:260] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:26.134155 28080 raft_consensus.cc:3058] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:26.129360 28154 raft_consensus.cc:1238] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20250905 20:00:26.098177 31099 raft_consensus.cc:2747] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:26.004146 31084 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:26.137988 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:26.138248 28122 leader_election.cc:336] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.781s (SENT)
W20250905 20:00:26.138571 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 110774us
I20250905 20:00:26.144035 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:26.145407 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:26.147783 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 59225.6us
I20250905 20:00:26.064711 31216 raft_consensus.cc:513] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:26.150506 28159 raft_consensus.cc:2466] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 7.
W20250905 20:00:26.158901 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:26.126722 31188 raft_consensus.cc:2747] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20250905 20:00:26.168210 28122 leader_election.cc:336] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.940s (SENT)
I20250905 20:00:26.169036 28122 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:26.169549 28123 leader_election.cc:304] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:26.137908 31196 leader_election.cc:290] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:26.099932 31208 consensus_queue.cc:1035] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.053s
I20250905 20:00:26.175022 28123 leader_election.cc:304] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
W20250905 20:00:26.177038 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:26.179374 28125 leader_election.cc:336] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.503s (SENT)
I20250905 20:00:26.180018 28125 leader_election.cc:304] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:26.180119 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:26.180994 28750 raft_consensus.cc:2466] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
I20250905 20:00:26.182343 28076 raft_consensus.cc:2366] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 6. Current term is 7.
I20250905 20:00:26.183743 28123 leader_election.cc:304] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:26.187224 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:26.152993 28746 raft_consensus.cc:3058] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Advancing to term 9
I20250905 20:00:26.185497 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 20:00:26.191303 28122 leader_election.cc:336] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.967s (SENT)
W20250905 20:00:26.192554 28122 leader_election.cc:336] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 1.967s (SENT)
I20250905 20:00:26.162971 31234 raft_consensus.cc:2802] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 10 FOLLOWER]: Leader election won for term 10
I20250905 20:00:26.100731 31266 raft_consensus.cc:513] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:26.197008 28122 leader_election.cc:336] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 2.032s (SENT)
W20250905 20:00:26.199640 31241 log.cc:927] Time spent T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 1.186s user 0.006s sys 0.000s
I20250905 20:00:26.201056 31216 leader_election.cc:290] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:26.201983 31234 raft_consensus.cc:695] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [term 10 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:26.203310 31234 consensus_queue.cc:237] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:26.206780 28122 leader_election.cc:336] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 2.055s (SENT)
W20250905 20:00:26.207484 28122 leader_election.cc:336] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 2.061s (SENT)
I20250905 20:00:26.202000 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:26.217271 28154 raft_consensus.cc:3058] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Advancing to term 7
W20250905 20:00:26.219174 28125 leader_election.cc:336] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.811s (SENT)
I20250905 20:00:26.219857 28125 leader_election.cc:304] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
W20250905 20:00:26.221434 28125 leader_election.cc:336] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 2.107s (SENT)
W20250905 20:00:26.222324 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:26.222991 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:26.225414 28122 leader_election.cc:304] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:26.223201 31087 raft_consensus.cc:2802] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 20:00:26.226578 31087 raft_consensus.cc:491] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:26.227134 31087 raft_consensus.cc:3058] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:26.224030 31190 raft_consensus.cc:2747] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250905 20:00:26.222906 31178 raft_consensus.cc:2747] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250905 20:00:26.232147 31269 raft_consensus.cc:2802] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election won for term 6
I20250905 20:00:26.232854 31196 consensus_queue.cc:1035] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 20:00:26.234949 31190 consensus_queue.cc:1046] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20250905 20:00:26.236585 31178 raft_consensus.cc:3053] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 LEADER]: Stepping down as leader of term 9
I20250905 20:00:26.237022 31178 raft_consensus.cc:738] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:26.238071 31178 consensus_queue.cc:260] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:26.239755 28154 raft_consensus.cc:2466] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 7.
I20250905 20:00:26.243093 28050 leader_election.cc:304] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:26.244393 31196 raft_consensus.cc:2802] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader election won for term 7
I20250905 20:00:26.222012 28125 leader_election.cc:304] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:26.245059 31196 raft_consensus.cc:695] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:26.246416 31196 consensus_queue.cc:237] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:26.186775 31099 raft_consensus.cc:2802] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250905 20:00:26.253203 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:26.260334 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:26.262645 28154 raft_consensus.cc:3058] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:26.189512 31177 raft_consensus.cc:2802] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader election won for term 7
W20250905 20:00:26.262058 27973 connection.cc:232] RPC call timeout handler was delayed by -3.71171912300008s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15400000000000003s for a response
I20250905 20:00:26.262504 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 20:00:26.265232 27973 connection.cc:232] RPC call timeout handler was delayed by -3.6964191230000552s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1557s for a response
I20250905 20:00:26.265226 28079 raft_consensus.cc:2408] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250905 20:00:26.129577 31263 raft_consensus.cc:513] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:26.265674 27973 connection.cc:232] RPC call timeout handler was delayed by -3.6649191230000042s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1592s for a response
W20250905 20:00:26.267890 27973 connection.cc:232] RPC call timeout handler was delayed by -3.458819123000012s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1821s for a response
W20250905 20:00:26.268272 27973 connection.cc:232] RPC call timeout handler was delayed by -3.3616191230000823s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19290000000000002s for a response
W20250905 20:00:26.268566 27973 connection.cc:232] RPC call timeout handler was delayed by -2.3311191230000077s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3074s for a response
I20250905 20:00:26.268618 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:26.170926 31151 consensus_queue.cc:1035] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.082s
I20250905 20:00:26.270262 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:26.254169 31193 raft_consensus.cc:491] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:26.216313 31261 raft_consensus.cc:2747] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250905 20:00:26.262248 31099 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:26.260214 30777 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:26.289933 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:26.290467 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 146802us
W20250905 20:00:26.291838 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 246383us
I20250905 20:00:26.308456 28085 raft_consensus.cc:2466] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
I20250905 20:00:26.308632 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:26.314303 28715 leader_election.cc:336] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 2.019s (SENT)
W20250905 20:00:26.319052 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 164636us
I20250905 20:00:26.322175 28079 raft_consensus.cc:2391] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 7: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:26.322789 28009 raft_consensus.cc:1238] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 7. Current term is 8. Ops: []
I20250905 20:00:26.323709 27973 leader_election.cc:304] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:26.216526 31171 consensus_queue.cc:1035] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.067s
I20250905 20:00:26.329213 28159 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
I20250905 20:00:26.354516 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:26.360519 28154 raft_consensus.cc:2466] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 7.
W20250905 20:00:26.374534 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:26.309788 31099 raft_consensus.cc:3058] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Advancing to term 9
W20250905 20:00:26.362957 28010 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1591) took 2938 ms (client timeout 2344 ms). Trace:
W20250905 20:00:26.379110 28010 rpcz_store.cc:269] 0905 20:00:23.383399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:23.383573 (+ 174us) service_pool.cc:225] Handling call
0905 20:00:26.321685 (+2938112us) spinlock_profiling.cc:243] Waited 2.18 s on lock 0x7b540034bf54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:26.321901 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5012797184}
I20250905 20:00:26.355259 28079 raft_consensus.cc:2466] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 7.
W20250905 20:00:26.369585 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:26.287695 31174 consensus_queue.cc:1035] T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.074s
W20250905 20:00:26.373581 28715 leader_election.cc:336] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193): Timed out: RequestConsensusVote RPC to 127.26.58.130:34193 timed out after 2.240s (SENT)
I20250905 20:00:26.128911 31282 consensus_queue.cc:1035] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.069s
I20250905 20:00:26.377516 28154 raft_consensus.cc:1238] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
I20250905 20:00:26.365641 31269 raft_consensus.cc:695] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:26.286811 31193 raft_consensus.cc:513] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:26.386232 28715 leader_election.cc:304] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:26.376056 28050 leader_election.cc:304] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:26.171875 31250 consensus_queue.cc:1035] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.081s
I20250905 20:00:26.362990 31171 consensus_queue.cc:1230] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer 7e9c9f591f05464f9b097bf003d14f4a log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20250905 20:00:26.395315 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 20:00:26.418844 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 45314us
W20250905 20:00:26.398152 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 163597us
W20250905 20:00:26.417520 31281 log.cc:927] Time spent T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.156s user 0.001s sys 0.000s
W20250905 20:00:26.432885 28077 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1653) took 2265 ms (client timeout 1812 ms). Trace:
I20250905 20:00:26.428557 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:26.454989 28080 raft_consensus.cc:1273] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20250905 20:00:26.455343 28077 rpcz_store.cc:269] 0905 20:00:24.167870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:24.168020 (+ 150us) service_pool.cc:225] Handling call
0905 20:00:26.432616 (+2264596us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b540023c654. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:26.432759 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2761564928}
I20250905 20:00:26.456079 28076 raft_consensus.cc:2408] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20250905 20:00:26.431201 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:26.457176 28084 raft_consensus.cc:2408] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20250905 20:00:26.425403 28156 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39586 (request call id 406) took 2264 ms (client timeout 2006 ms). Trace:
W20250905 20:00:26.457674 28156 rpcz_store.cc:269] 0905 20:00:24.160684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:24.161096 (+ 412us) service_pool.cc:225] Handling call
0905 20:00:26.425160 (+2264064us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b5400200b54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:26.425320 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2559538048}
W20250905 20:00:26.441074 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 189766us
I20250905 20:00:26.321911 30777 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:26.388862 31285 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:26.357024 31261 consensus_queue.cc:1046] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.042s
W20250905 20:00:26.443107 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 67170.6us
I20250905 20:00:26.435257 31238 raft_consensus.cc:491] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:26.447358 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 160837us
W20250905 20:00:26.450897 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:26.446601 31188 raft_consensus.cc:2802] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader election won for term 7
W20250905 20:00:26.452005 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 273792us
I20250905 20:00:26.454229 28079 raft_consensus.cc:2391] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 8: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:26.421379 31275 raft_consensus.cc:2747] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250905 20:00:26.422765 31263 leader_election.cc:290] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:26.464885 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:26.496011 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:26.496539 27973 leader_election.cc:304] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:26.454722 31234 raft_consensus.cc:2747] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250905 20:00:26.488022 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:26.463110 31267 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:26.499164 28084 raft_consensus.cc:2391] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 7: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 20:00:26.427682 31087 raft_consensus.cc:513] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:26.471490 31227 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20250905 20:00:26.468920 31285 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:26.517549 31276 raft_consensus.cc:2747] T e707b0f561934674947fb463ce626a07 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:26.530566 28158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2299) took 1258 ms. Trace:
I20250905 20:00:26.522249 27973 leader_election.cc:304] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:26.521145 28077 raft_consensus.cc:1238] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 4. Current term is 5. Ops: []
W20250905 20:00:26.556629 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 52714.8us
I20250905 20:00:26.562561 28076 raft_consensus.cc:1238] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 LEADER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:26.562795 28005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 2168) took 1256 ms. Trace:
I20250905 20:00:26.541003 28084 raft_consensus.cc:1238] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [term 5 LEADER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:26.566920 28077 raft_consensus.cc:1238] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 4. Current term is 5. Ops: []
W20250905 20:00:26.569188 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 477820us
W20250905 20:00:26.571204 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 49045.7us
I20250905 20:00:26.530719 31282 raft_consensus.cc:2802] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 20:00:26.571277 28150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1737) took 1424 ms. Trace:
I20250905 20:00:26.520411 31213 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:26.590665 28150 rpcz_store.cc:276] 0905 20:00:25.147006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:25.736901 (+589895us) service_pool.cc:225] Handling call
0905 20:00:25.763758 (+ 26857us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:25.765616 (+ 1858us) raft_consensus.cc:1532] Early marking committed up to index 5
0905 20:00:25.803848 (+ 38232us) spinlock_profiling.cc:243] Waited 4.18 ms on lock 0x7b18001690c0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49ac86d47 00007fe4a38dcb99 00007fe49aca369b 00007fe49aca193f 00007fe49aca1455 00007fe4a1605337 00007fe49dc23957 00007fe49dc3306f 00007fe49dc32fe8
0905 20:00:26.094671 (+290823us) log.cc:844] Serialized 14 byte log entry
0905 20:00:26.469723 (+375052us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:26.479348 (+ 9625us) log.cc:844] Serialized 26 byte log entry
0905 20:00:26.480392 (+ 1044us) raft_consensus.cc:1643] Marking committed up to 5
0905 20:00:26.480448 (+ 56us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:26.481134 (+ 686us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:26.565217 (+ 84083us) raft_consensus.cc:1683] finished
0905 20:00:26.570958 (+ 5741us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:26.571183 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1384,"prepare.queue_time_us":395787,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":55,"raft.queue_time_us":258641,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":8548,"spinlock_wait_cycles":9618816,"thread_start_us":621186,"threads_started":3,"wal-append.queue_time_us":288904}
I20250905 20:00:26.602602 28051 leader_election.cc:304] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:26.563357 28005 rpcz_store.cc:276] 0905 20:00:25.306353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:25.631812 (+325459us) service_pool.cc:225] Handling call
0905 20:00:25.678650 (+ 46838us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:25.762320 (+ 83670us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:25.762362 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:25.797374 (+ 35012us) log.cc:844] Serialized 26 byte log entry
0905 20:00:26.369638 (+572264us) raft_consensus.cc:1643] Marking committed up to 5
0905 20:00:26.399718 (+ 30080us) log.cc:844] Serialized 14 byte log entry
0905 20:00:26.562314 (+162596us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:26.562384 (+ 70us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:26.562446 (+ 62us) raft_consensus.cc:1683] finished
0905 20:00:26.562461 (+ 15us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:26.562724 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":29951,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"raft.queue_time_us":282529,"raft.run_cpu_time_us":317,"raft.run_wall_time_us":44851,"thread_start_us":553405,"threads_started":3,"wal-append.queue_time_us":668784}
I20250905 20:00:26.555974 31188 raft_consensus.cc:695] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [term 7 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:26.488962 31238 raft_consensus.cc:513] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:26.553567 28158 rpcz_store.cc:276] 0905 20:00:25.272117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:25.992439 (+720322us) service_pool.cc:225] Handling call
0905 20:00:26.032449 (+ 40010us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:26.080361 (+ 47912us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:26.080393 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:26.080856 (+ 463us) log.cc:844] Serialized 26 byte log entry
0905 20:00:26.291405 (+210549us) raft_consensus.cc:1643] Marking committed up to 2
0905 20:00:26.291473 (+ 68us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:26.291575 (+ 102us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:26.530207 (+238632us) raft_consensus.cc:1683] finished
0905 20:00:26.530245 (+ 38us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:26.530471 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14621,"raft.queue_time_us":61163,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":2709,"thread_start_us":200492,"threads_started":1,"wal-append.queue_time_us":396904}
I20250905 20:00:26.539691 31263 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:26.556811 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 226196us
I20250905 20:00:26.599499 28076 raft_consensus.cc:1238] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:26.500854 31261 consensus_queue.cc:1035] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.069s
I20250905 20:00:26.521891 31267 raft_consensus.cc:2747] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:26.615634 28077 raft_consensus.cc:1273] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20250905 20:00:26.576031 28084 raft_consensus.cc:1273] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250905 20:00:26.523411 31275 consensus_queue.cc:1035] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.077s
I20250905 20:00:26.618569 28051 leader_election.cc:304] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:26.518342 31083 raft_consensus.cc:513] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:26.590036 31276 raft_consensus.cc:2802] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 20:00:26.625934 28051 leader_election.cc:304] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:26.561357 31187 consensus_queue.cc:1046] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20250905 20:00:26.590474 31282 raft_consensus.cc:491] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:26.525686 31250 consensus_queue.cc:1035] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.092s
W20250905 20:00:26.650938 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:26.620007 31227 raft_consensus.cc:2802] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 20:00:26.526777 31174 consensus_queue.cc:1035] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.090s
I20250905 20:00:26.633009 28051 leader_election.cc:304] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:26.635274 28750 raft_consensus.cc:1273] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250905 20:00:26.637494 27973 leader_election.cc:304] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:26.496246 31269 consensus_queue.cc:237] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:26.637893 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 147057us
I20250905 20:00:26.629544 28152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2302) took 1036 ms. Trace:
I20250905 20:00:26.628495 30777 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:26.630148 31193 leader_election.cc:290] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:26.630666 31221 raft_consensus.cc:2747] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250905 20:00:26.629889 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 125767us
I20250905 20:00:26.661821 28152 rpcz_store.cc:276] 0905 20:00:25.593572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:26.020773 (+427201us) service_pool.cc:225] Handling call
0905 20:00:26.079128 (+ 58355us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:26.079759 (+ 631us) raft_consensus.cc:1532] Early marking committed up to index 4
0905 20:00:26.262413 (+182654us) log.cc:844] Serialized 14 byte log entry
0905 20:00:26.629155 (+366742us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:26.629214 (+ 59us) raft_consensus.cc:1643] Marking committed up to 4
0905 20:00:26.629263 (+ 49us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:26.629333 (+ 70us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:26.629468 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":186265,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":62,"raft.queue_time_us":74957,"raft.run_cpu_time_us":54,"raft.run_wall_time_us":56,"thread_start_us":529408,"threads_started":3,"wal-append.queue_time_us":370200}
I20250905 20:00:26.672838 28157 raft_consensus.cc:1273] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250905 20:00:26.695242 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:26.696346 28077 raft_consensus.cc:2466] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 7.
W20250905 20:00:26.668195 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 192665us
I20250905 20:00:26.636324 31276 raft_consensus.cc:491] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:26.619228 31216 consensus_queue.cc:1035] T bbd67b7a3c3244838ec2d727f746cc0b P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
I20250905 20:00:26.644426 31213 raft_consensus.cc:2747] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250905 20:00:26.630275 31306 raft_consensus.cc:491] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:26.648185 31282 raft_consensus.cc:3058] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:26.708974 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:26.656627 31227 raft_consensus.cc:491] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:26.712396 28150 raft_consensus.cc:1273] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250905 20:00:26.630460 31263 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:26.743582 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:26.743775 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:26.744341 28748 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
I20250905 20:00:26.744727 28125 leader_election.cc:304] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 20:00:26.725847 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 88456.9us
I20250905 20:00:26.673378 31220 raft_consensus.cc:2747] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:26.737303 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:26.741376 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:26.746675 27973 leader_election.cc:304] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:26.747591 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:26.751425 28717 leader_election.cc:304] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:26.718173 31276 raft_consensus.cc:3058] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:26.751473 28076 raft_consensus.cc:2466] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
I20250905 20:00:26.749202 28745 pending_rounds.cc:85] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a: Aborting all ops after (but not including) 5
I20250905 20:00:26.753931 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:26.769709 28745 pending_rounds.cc:107] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a: Aborting uncommitted NO_OP operation due to leader change: 5.6
W20250905 20:00:26.756609 28125 leader_election.cc:336] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.717s (SENT)
W20250905 20:00:26.745219 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 114431us
I20250905 20:00:26.771099 28125 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:26.772984 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:26.773870 28150 raft_consensus.cc:2408] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 11 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20250905 20:00:26.784698 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:26.785404 28076 raft_consensus.cc:2466] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 20:00:26.770193 28745 raft_consensus.cc:2887] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250905 20:00:26.788456 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:26.745203 31227 raft_consensus.cc:3058] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:26.724447 31224 consensus_queue.cc:1035] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.055s
I20250905 20:00:26.793262 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:26.722812 31193 consensus_queue.cc:1035] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
W20250905 20:00:26.793103 28125 leader_election.cc:336] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.781s (SENT)
W20250905 20:00:26.797924 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:26.800540 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:26.767050 31261 raft_consensus.cc:2747] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:26.809513 28125 leader_election.cc:304] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:26.816457 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:26.794308 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:26.818238 28748 raft_consensus.cc:2391] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 7: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:26.772233 31151 raft_consensus.cc:2802] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 20:00:26.810014 28747 raft_consensus.cc:3058] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:26.671831 31311 consensus_queue.cc:1035] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.135s
I20250905 20:00:26.789206 31087 leader_election.cc:290] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:26.821640 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:26.799577 31083 leader_election.cc:290] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:26.809567 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 357101us
I20250905 20:00:26.829294 28751 raft_consensus.cc:1273] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20250905 20:00:26.835403 31304 log.cc:927] Time spent T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.152s user 0.000s sys 0.002s
I20250905 20:00:26.835767 31315 raft_consensus.cc:2747] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250905 20:00:26.833952 28125 leader_election.cc:336] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.879s (SENT)
I20250905 20:00:26.841552 28125 leader_election.cc:304] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 20:00:26.846544 28125 leader_election.cc:336] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.940s (SENT)
I20250905 20:00:26.848508 28122 leader_election.cc:304] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
W20250905 20:00:26.851457 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:26.877842 28125 leader_election.cc:336] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.967s (SENT)
I20250905 20:00:26.878187 28747 raft_consensus.cc:2466] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 8.
I20250905 20:00:26.878469 28125 leader_election.cc:304] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:26.881201 28715 leader_election.cc:304] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:26.734102 31188 consensus_queue.cc:237] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:26.866266 28122 leader_election.cc:304] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:26.879014 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:26.802872 31312 raft_consensus.cc:2802] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 20:00:26.745400 31306 raft_consensus.cc:513] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:26.888695 28123 leader_election.cc:304] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:26.823393 31151 raft_consensus.cc:491] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:26.896054 28748 raft_consensus.cc:1273] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250905 20:00:26.889647 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:26.787014 31302 consensus_queue.cc:1035] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.112s
I20250905 20:00:26.717706 31293 consensus_queue.cc:1035] T 4414c08fe910403b80af2d558aac36b2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.113s
I20250905 20:00:26.789848 31099 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:26.887846 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:26.904577 28077 raft_consensus.cc:2466] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 8.
I20250905 20:00:26.804481 31196 consensus_queue.cc:1035] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.083s
W20250905 20:00:26.897797 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 127025us
I20250905 20:00:26.882580 31316 raft_consensus.cc:2747] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:26.897974 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:26.911221 28155 raft_consensus.cc:1273] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250905 20:00:26.912231 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:26.913410 31250 raft_consensus.cc:2802] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 20:00:26.832806 31171 raft_consensus.cc:2747] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250905 20:00:26.879648 31315 raft_consensus.cc:2747] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 20:00:26.913630 28715 leader_election.cc:304] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:26.917546 28715 leader_election.cc:304] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:26.890111 31312 raft_consensus.cc:491] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:26.920975 28150 raft_consensus.cc:1238] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 7. Current term is 8. Ops: []
W20250905 20:00:26.921954 28746 consensus_meta.cc:287] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.662s user 0.003s sys 0.009s
I20250905 20:00:26.922883 31325 raft_consensus.cc:2802] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 20:00:26.923396 31325 raft_consensus.cc:491] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:26.923836 31228 raft_consensus.cc:3053] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 20:00:26.924213 31229 raft_consensus.cc:491] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:26.924501 31228 raft_consensus.cc:738] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:26.924854 31193 raft_consensus.cc:2802] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader election won for term 8
I20250905 20:00:26.924818 31229 raft_consensus.cc:513] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:26.925523 31193 raft_consensus.cc:695] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:26.925391 31228 consensus_queue.cc:260] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:26.926658 31193 consensus_queue.cc:237] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:26.927551 31229 leader_election.cc:290] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:26.889672 31238 leader_election.cc:290] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:26.883924 31285 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:26.901387 28125 leader_election.cc:336] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 2.032s (SENT)
I20250905 20:00:26.809067 31300 raft_consensus.cc:2747] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20250905 20:00:26.903077 28122 leader_election.cc:304] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:26.931141 28125 leader_election.cc:304] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:26.820600 28743 raft_consensus.cc:1238] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:26.887252 31261 consensus_queue.cc:1035] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20250905 20:00:26.923909 31325 raft_consensus.cc:3058] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:26.914194 31250 raft_consensus.cc:491] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:26.902513 31151 raft_consensus.cc:3058] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:26.957791 31317 raft_consensus.cc:491] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:26.940914 28715 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:26.900213 31087 raft_consensus.cc:2747] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250905 20:00:26.951403 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:26.958936 28005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:26.959743 28077 raft_consensus.cc:2466] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 8.
I20250905 20:00:26.960409 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:26.960919 28004 raft_consensus.cc:2466] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 6.
W20250905 20:00:26.954474 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:26.948022 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 201386us
I20250905 20:00:26.952656 31250 raft_consensus.cc:3058] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Advancing to term 8
W20250905 20:00:26.964031 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:26.971084 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:26.965005 28004 raft_consensus.cc:3053] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Stepping down as leader of term 5
I20250905 20:00:26.965083 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:26.940589 31228 raft_consensus.cc:3058] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [term 1 FOLLOWER]: Advancing to term 2
I20250905 20:00:26.972235 28002 raft_consensus.cc:2466] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
I20250905 20:00:26.971886 28004 raft_consensus.cc:738] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:26.921010 31312 raft_consensus.cc:3058] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:26.980966 28008 raft_consensus.cc:1273] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250905 20:00:26.977968 27973 leader_election.cc:304] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:26.931550 28746 raft_consensus.cc:2466] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 9.
I20250905 20:00:26.952529 31171 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:26.979741 28010 raft_consensus.cc:1273] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250905 20:00:26.986380 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:26.986311 28051 leader_election.cc:304] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:26.990656 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:26.981197 28004 consensus_queue.cc:260] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:26.994542 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:26.993078 28007 raft_consensus.cc:1273] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250905 20:00:26.993402 28008 raft_consensus.cc:3058] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:26.992779 28003 raft_consensus.cc:2391] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 8: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:26.993873 28051 leader_election.cc:304] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:26.998517 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:26.999191 28003 raft_consensus.cc:2466] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
I20250905 20:00:26.999437 31187 raft_consensus.cc:2802] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:26.999852 31187 raft_consensus.cc:491] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:27.000123 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:27.000207 31187 raft_consensus.cc:3058] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:27.001122 28003 raft_consensus.cc:2391] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 10: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:27.001281 31212 consensus_queue.cc:1035] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250905 20:00:27.000238 30777 consensus_queue.cc:1035] T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 20:00:27.000485 31221 consensus_queue.cc:1035] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
W20250905 20:00:26.993427 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 49477.2us
I20250905 20:00:27.005929 28715 leader_election.cc:304] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:27.007165 31238 raft_consensus.cc:2747] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250905 20:00:26.991487 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:27.009095 28002 raft_consensus.cc:2391] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 7: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:27.008911 28715 leader_election.cc:304] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:27.014530 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:27.016567 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:27.022579 31305 raft_consensus.cc:2747] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:27.024549 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:27.025883 31326 log.cc:927] Time spent T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20250905 20:00:27.026492 31252 raft_consensus.cc:2762] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20250905 20:00:27.027951 31328 consensus_queue.cc:1046] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20250905 20:00:27.028976 31328 raft_consensus.cc:3053] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 LEADER]: Stepping down as leader of term 7
I20250905 20:00:27.029343 31328 raft_consensus.cc:738] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:27.030567 28007 raft_consensus.cc:2391] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 7: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:27.031543 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:27.034379 28007 raft_consensus.cc:3058] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:27.032716 28051 leader_election.cc:304] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:26.978502 31210 raft_consensus.cc:491] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:26.977095 31316 raft_consensus.cc:2747] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:27.040058 28004 raft_consensus.cc:3058] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:27.045328 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:27.037519 28051 leader_election.cc:304] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:27.050829 31174 raft_consensus.cc:2747] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250905 20:00:27.051357 31306 leader_election.cc:290] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:27.054337 28051 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:27.058125 28010 raft_consensus.cc:2408] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250905 20:00:27.053743 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:27.061079 28715 leader_election.cc:304] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
W20250905 20:00:27.060141 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 263946us
I20250905 20:00:26.990761 31314 raft_consensus.cc:2802] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:27.062495 28051 leader_election.cc:304] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:27.085167 28008 raft_consensus.cc:2466] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 7.
I20250905 20:00:27.039858 31210 raft_consensus.cc:3058] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Advancing to term 5
W20250905 20:00:27.061707 31240 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20250905 20:00:27.093757 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:27.066773 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:27.040524 31216 raft_consensus.cc:2747] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250905 20:00:27.042711 31238 raft_consensus.cc:2747] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250905 20:00:27.047498 31263 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:26.971201 31317 raft_consensus.cc:513] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:27.066087 31252 raft_consensus.cc:2802] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:27.019771 31171 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:27.119660 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:27.111370 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:27.116706 28077 raft_consensus.cc:2466] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
W20250905 20:00:27.123539 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 222167us
I20250905 20:00:27.061431 31229 raft_consensus.cc:2747] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250905 20:00:27.075047 31314 raft_consensus.cc:491] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
W20250905 20:00:27.184659 28125 leader_election.cc:336] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 2.055s (SENT)
I20250905 20:00:27.194060 28004 raft_consensus.cc:1273] T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250905 20:00:27.193794 28125 leader_election.cc:304] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:27.186093 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:27.187119 28010 raft_consensus.cc:2391] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 7: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
W20250905 20:00:27.181425 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:27.204550 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:27.205469 28004 raft_consensus.cc:2408] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20250905 20:00:27.074311 31196 consensus_queue.cc:1035] T 10dd790c7b5c4cc5978c3545b5a06868 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.109s
I20250905 20:00:27.206516 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:27.207456 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:27.208359 28125 leader_election.cc:336] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 2.061s (SENT)
I20250905 20:00:27.127130 31252 raft_consensus.cc:491] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:27.208832 28125 leader_election.cc:304] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:27.208559 28010 raft_consensus.cc:2374] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 7. Re-sending same reply.
I20250905 20:00:27.212268 28004 raft_consensus.cc:2391] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 6: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:27.215794 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:27.217651 28004 raft_consensus.cc:2408] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20250905 20:00:27.101305 31295 raft_consensus.cc:2747] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250905 20:00:27.224385 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:27.221982 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:27.220443 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 111184us
I20250905 20:00:27.197350 31238 raft_consensus.cc:2747] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 20:00:27.207167 31334 raft_consensus.cc:2747] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250905 20:00:27.225755 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 188197us
W20250905 20:00:27.226357 28125 leader_election.cc:336] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 2.108s (SENT)
I20250905 20:00:27.030205 31328 consensus_queue.cc:260] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:27.228098 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:27.228353 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:27.229353 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:27.198799 31314 raft_consensus.cc:3058] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:27.230933 28125 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:27.208163 31277 raft_consensus.cc:3058] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 1 FOLLOWER]: Advancing to term 2
I20250905 20:00:27.231127 28715 leader_election.cc:304] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:27.224231 28743 raft_consensus.cc:1238] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20250905 20:00:27.221321 31252 raft_consensus.cc:3058] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:27.203737 31099 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:27.250370 28049 leader_election.cc:304] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:27.252635 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:27.253491 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:27.254181 28077 raft_consensus.cc:2466] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 20:00:27.253427 28010 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
I20250905 20:00:27.256053 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:27.256768 28747 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:27.231487 31316 raft_consensus.cc:2747] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250905 20:00:27.236340 31084 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:27.273741 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:27.274996 28751 raft_consensus.cc:2466] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 8.
I20250905 20:00:27.276279 28009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 378) took 3143 ms. Trace:
I20250905 20:00:27.276856 28009 rpcz_store.cc:276] 0905 20:00:24.132792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:24.132979 (+ 187us) service_pool.cc:225] Handling call
0905 20:00:26.322613 (+2189634us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:26.323520 (+ 907us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:27.269831 (+946311us) spinlock_profiling.cc:243] Waited 2.17 s on lock 0x7b540034bf50. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:27.276204 (+ 6373us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4982244608}
I20250905 20:00:27.283725 28010 raft_consensus.cc:1273] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250905 20:00:27.283545 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:27.284798 28077 raft_consensus.cc:2466] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
I20250905 20:00:27.290633 28744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40918 (request call id 653) took 1988 ms. Trace:
I20250905 20:00:27.291312 28744 rpcz_store.cc:276] 0905 20:00:25.302745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:25.302923 (+ 178us) service_pool.cc:225] Handling call
0905 20:00:25.364031 (+ 61108us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:25.820743 (+456712us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:27.290228 (+1469485us) spinlock_profiling.cc:243] Waited 437 ms on lock 0x7b54005bed54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:27.290507 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1004472576}
I20250905 20:00:27.255385 31261 raft_consensus.cc:2747] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250905 20:00:27.255270 31328 raft_consensus.cc:3058] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:27.320545 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:27.301016 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:27.327422 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:27.321521 28084 raft_consensus.cc:2466] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 7.
I20250905 20:00:27.329057 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:27.330149 28084 raft_consensus.cc:2466] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
I20250905 20:00:27.331470 28747 raft_consensus.cc:3058] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Advancing to term 9
I20250905 20:00:27.335356 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:27.311371 31333 log.cc:927] Time spent T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.055s user 0.000s sys 0.001s
W20250905 20:00:27.323329 31308 log.cc:927] Time spent T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.332s user 0.000s sys 0.002s
W20250905 20:00:27.306378 31329 log.cc:927] Time spent T 24cf07baf43247b6bf63063106b79836 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.099s user 0.000s sys 0.001s
I20250905 20:00:27.339849 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:27.351701 28077 raft_consensus.cc:2466] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 9.
W20250905 20:00:27.355877 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 99842.6us
W20250905 20:00:27.356900 28125 leader_election.cc:336] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 2.237s (SENT)
I20250905 20:00:27.307158 31317 leader_election.cc:290] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:27.306325 31195 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:27.359993 28125 leader_election.cc:304] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:27.361084 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:27.363317 28005 raft_consensus.cc:2466] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 8.
W20250905 20:00:27.345368 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 381171us
I20250905 20:00:27.376325 31171 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:27.366708 31195 raft_consensus.cc:2747] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250905 20:00:27.442054 28085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1691) took 1316 ms (client timeout 1525 ms). Trace:
W20250905 20:00:27.442973 28085 rpcz_store.cc:269] 0905 20:00:26.125794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:26.143723 (+ 17929us) service_pool.cc:225] Handling call
0905 20:00:27.398752 (+1255029us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b54000f4454. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:27.441992 (+ 43240us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":364258048}
I20250905 20:00:27.446767 27973 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:27.475244 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:27.478562 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:27.378191 31276 consensus_meta.cc:287] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.590s user 0.013s sys 0.000s
I20250905 20:00:27.479683 28157 raft_consensus.cc:2466] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
W20250905 20:00:27.482016 31337 log.cc:927] Time spent T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.188s user 0.003s sys 0.003s
I20250905 20:00:27.483405 30978 raft_consensus.cc:491] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:27.502779 28150 raft_consensus.cc:1238] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 1. Current term is 2. Ops: []
W20250905 20:00:27.487102 28151 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39586 (request call id 405) took 3327 ms (client timeout 1976 ms). Trace:
W20250905 20:00:27.503700 28151 rpcz_store.cc:269] 0905 20:00:24.160128 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:24.160294 (+ 166us) service_pool.cc:225] Handling call
0905 20:00:27.486892 (+3326598us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54000e1ad4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:27.487020 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2398968064}
I20250905 20:00:27.488013 28158 raft_consensus.cc:1273] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20250905 20:00:27.488480 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 131626us
I20250905 20:00:27.504842 28154 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2313) took 1263 ms. Trace:
W20250905 20:00:27.490556 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 43878.1us
I20250905 20:00:27.493479 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:27.505367 28154 rpcz_store.cc:276] 0905 20:00:26.241339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:26.390866 (+149527us) service_pool.cc:225] Handling call
0905 20:00:26.391344 (+ 478us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:26.391804 (+ 460us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:26.391844 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:26.392863 (+ 1019us) log.cc:844] Serialized 26 byte log entry
0905 20:00:26.617365 (+224502us) raft_consensus.cc:1643] Marking committed up to 3
0905 20:00:26.617525 (+ 160us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:26.617630 (+ 105us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:27.504629 (+886999us) raft_consensus.cc:1683] finished
0905 20:00:27.504649 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:27.504786 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":131120,"raft.run_cpu_time_us":51,"raft.run_wall_time_us":52,"thread_start_us":224090,"threads_started":1,"wal-append.queue_time_us":556766}
I20250905 20:00:27.507009 28751 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 11.
I20250905 20:00:27.507489 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:27.502765 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:27.508278 28154 raft_consensus.cc:2391] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 9: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:27.508607 27974 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:27.509061 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:27.509850 28154 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 10.
W20250905 20:00:27.515148 31336 log.cc:927] Time spent T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.156s user 0.000s sys 0.001s
W20250905 20:00:27.519364 31282 consensus_meta.cc:287] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.768s user 0.010s sys 0.003s
I20250905 20:00:27.519923 31282 raft_consensus.cc:513] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:27.527566 28008 raft_consensus.cc:2366] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 7. Current term is 8.
I20250905 20:00:27.527884 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:27.528496 28154 raft_consensus.cc:3058] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:27.528795 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:27.530800 28125 leader_election.cc:336] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 2.391s (SENT)
I20250905 20:00:27.531523 28125 leader_election.cc:304] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:27.490024 31171 raft_consensus.cc:2747] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20250905 20:00:27.542936 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:27.546015 28125 leader_election.cc:336] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 2.411s (SENT)
I20250905 20:00:27.547407 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
W20250905 20:00:27.549212 28125 leader_election.cc:336] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 2.499s (SENT)
I20250905 20:00:27.549650 28125 leader_election.cc:304] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
W20250905 20:00:27.550225 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:27.553999 28150 raft_consensus.cc:1273] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20250905 20:00:27.554224 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:27.538761 27973 leader_election.cc:304] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:27.565798 31178 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Advancing to term 10
W20250905 20:00:27.566324 28007 consensus_meta.cc:287] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.530s user 0.004s sys 0.000s
I20250905 20:00:27.570538 28159 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33756 (request call id 2310) took 1365 ms. Trace:
I20250905 20:00:27.533274 31084 raft_consensus.cc:2802] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20250905 20:00:27.577464 28007 raft_consensus.cc:2466] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
I20250905 20:00:27.577518 28159 rpcz_store.cc:276] 0905 20:00:26.204785 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:26.281704 (+ 76919us) service_pool.cc:225] Handling call
0905 20:00:27.570299 (+1288595us) spinlock_profiling.cc:243] Waited 16.5 ms on lock 0x7b5400316354. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:27.570472 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":38048512}
I20250905 20:00:27.577051 28154 raft_consensus.cc:2408] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20250905 20:00:27.535370 31282 leader_election.cc:290] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:27.546921 31190 raft_consensus.cc:2747] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:27.582923 28747 raft_consensus.cc:2466] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 9.
I20250905 20:00:27.583631 31263 raft_consensus.cc:2747] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250905 20:00:27.584273 31187 consensus_meta.cc:287] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.581s user 0.007s sys 0.002s
W20250905 20:00:27.600621 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 51449.8us
I20250905 20:00:27.579433 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:27.586102 27975 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 20:00:27.602931 28125 leader_election.cc:336] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 2.917s (SENT)
I20250905 20:00:27.603528 28125 leader_election.cc:304] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 20:00:27.586763 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:27.605295 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 20:00:27.605770 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 355450us
I20250905 20:00:27.606371 28159 raft_consensus.cc:2391] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 7: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:27.602948 28150 raft_consensus.cc:2466] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 8.
I20250905 20:00:27.590970 28154 raft_consensus.cc:1238] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 7. Current term is 8. Ops: []
I20250905 20:00:27.526829 30978 raft_consensus.cc:513] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:27.594334 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 55757.1us
I20250905 20:00:27.598452 28123 leader_election.cc:304] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
W20250905 20:00:27.599776 31331 log.cc:927] Time spent T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.258s user 0.003s sys 0.004s
I20250905 20:00:27.510794 31276 raft_consensus.cc:513] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:27.600786 28078 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50660 (request call id 353) took 3469 ms. Trace:
I20250905 20:00:27.619727 28078 rpcz_store.cc:276] 0905 20:00:24.125362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:24.125462 (+ 100us) service_pool.cc:225] Handling call
0905 20:00:24.142704 (+ 17242us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:26.069156 (+1926452us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:27.594148 (+1524992us) spinlock_profiling.cc:243] Waited 1.89 s on lock 0x7b54003a9654. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:27.594371 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4354057472}
I20250905 20:00:27.604189 27975 leader_election.cc:304] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:27.611186 27973 leader_election.cc:304] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:27.611454 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:27.579473 31084 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:27.626778 31084 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 11 FOLLOWER]: Advancing to term 12
I20250905 20:00:27.628646 28159 raft_consensus.cc:1273] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 10 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 8. (term mismatch)
I20250905 20:00:27.597544 31306 raft_consensus.cc:3058] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:27.600397 31171 raft_consensus.cc:2802] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:27.631685 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:27.666160 31171 raft_consensus.cc:491] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:27.666685 31171 raft_consensus.cc:3058] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:27.629607 28159 pending_rounds.cc:85] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting all ops after (but not including) 7
I20250905 20:00:27.631556 27975 leader_election.cc:304] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:27.662478 28004 raft_consensus.cc:2391] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 8: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:27.671137 28159 pending_rounds.cc:107] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20250905 20:00:27.671615 28159 raft_consensus.cc:2887] T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250905 20:00:27.673022 31276 leader_election.cc:290] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:27.663260 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:27.642830 28154 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
W20250905 20:00:27.675335 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:27.675191 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:27.622959 31196 raft_consensus.cc:3058] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:27.682142 30777 consensus_queue.cc:1035] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20250905 20:00:27.683099 28082 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1677) took 1851 ms (client timeout 1741 ms). Trace:
W20250905 20:00:27.683537 28082 rpcz_store.cc:269] 0905 20:00:25.832198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:25.835040 (+ 2842us) service_pool.cc:225] Handling call
0905 20:00:27.682859 (+1847819us) spinlock_profiling.cc:243] Waited 63.1 ms on lock 0x7b540019ebd4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:27.683023 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":145078144}
W20250905 20:00:27.687011 31330 log.cc:927] Time spent T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.487s user 0.010s sys 0.007s
I20250905 20:00:27.689004 28746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40926 (request call id 547) took 1922 ms. Trace:
I20250905 20:00:27.689515 28746 rpcz_store.cc:276] 0905 20:00:25.766771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:25.780587 (+ 13816us) service_pool.cc:225] Handling call
0905 20:00:27.688813 (+1908226us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b540048c6d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:27.688939 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":234535680}
I20250905 20:00:27.692466 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:27.693532 28081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1681) took 1759 ms. Trace:
I20250905 20:00:27.621413 31293 raft_consensus.cc:2802] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250905 20:00:27.694672 31293 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:27.695066 31293 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 10 FOLLOWER]: Advancing to term 11
W20250905 20:00:27.695458 31227 consensus_meta.cc:287] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.871s user 0.015s sys 0.001s
I20250905 20:00:27.697229 28080 raft_consensus.cc:1238] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:27.693920 28081 rpcz_store.cc:276] 0905 20:00:25.934097 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:25.937918 (+ 3821us) service_pool.cc:225] Handling call
0905 20:00:27.692287 (+1754369us) spinlock_profiling.cc:243] Waited 2.26 ms on lock 0x7b5800204f98. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b7db60 00007fe4a2b7d444 00007fe49c2050c4 00007fe49c205052
0905 20:00:27.693457 (+ 1170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5191936}
I20250905 20:00:27.680626 28049 leader_election.cc:304] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:27.681422 31216 consensus_queue.cc:1035] T 986c4f475a8d46f39465775db70416d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250905 20:00:27.599316 31210 raft_consensus.cc:513] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:27.631219 30946 raft_consensus.cc:2802] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Leader election won for term 9
I20250905 20:00:27.618955 31341 raft_consensus.cc:2762] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
W20250905 20:00:27.681953 31349 log.cc:927] Time spent T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.193s user 0.000s sys 0.001s
I20250905 20:00:27.631892 31190 raft_consensus.cc:2747] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:27.663200 31282 raft_consensus.cc:2802] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250905 20:00:27.689515 31325 consensus_meta.cc:287] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.727s user 0.007s sys 0.000s
I20250905 20:00:27.696061 31227 raft_consensus.cc:513] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:27.646224 31346 raft_consensus.cc:2747] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250905 20:00:27.716651 30777 raft_consensus.cc:2802] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250905 20:00:27.616528 31187 raft_consensus.cc:513] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:27.761497 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:27.757020 28752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40918 (request call id 660) took 1011 ms. Trace:
I20250905 20:00:27.646466 31339 consensus_queue.cc:1046] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.080s
I20250905 20:00:27.762836 28752 rpcz_store.cc:276] 0905 20:00:26.746030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:26.767875 (+ 21845us) service_pool.cc:225] Handling call
0905 20:00:26.797775 (+ 29900us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:26.802625 (+ 4850us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:26.802671 (+ 46us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:26.858229 (+ 55558us) log.cc:844] Serialized 26 byte log entry
0905 20:00:27.100553 (+242324us) raft_consensus.cc:1643] Marking committed up to 7
0905 20:00:27.100632 (+ 79us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:27.100738 (+ 106us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:27.731818 (+631080us) raft_consensus.cc:1683] finished
0905 20:00:27.731852 (+ 34us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:27.756954 (+ 25102us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12971,"raft.queue_time_us":4792,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":13139,"thread_start_us":194287,"threads_started":1,"wal-append.queue_time_us":302975}
I20250905 20:00:27.755890 31221 consensus_queue.cc:1046] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
I20250905 20:00:27.713835 31217 consensus_queue.cc:1046] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.044s
I20250905 20:00:27.724509 31284 raft_consensus.cc:491] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:27.733315 31282 raft_consensus.cc:491] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:27.770315 31188 consensus_queue.cc:1035] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
W20250905 20:00:27.793929 28125 leader_election.cc:336] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.546s (SENT)
I20250905 20:00:27.817291 28125 leader_election.cc:304] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:27.795794 28158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2320) took 1100 ms. Trace:
I20250905 20:00:27.818053 28158 rpcz_store.cc:276] 0905 20:00:26.695078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:26.695281 (+ 203us) service_pool.cc:225] Handling call
0905 20:00:27.487576 (+792295us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:27.504713 (+ 17137us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:27.795427 (+290714us) spinlock_profiling.cc:243] Waited 792 ms on lock 0x7b54000e1ad0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:27.795671 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1821059328}
W20250905 20:00:27.818609 28125 leader_election.cc:336] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.554s (SENT)
I20250905 20:00:27.820132 28125 leader_election.cc:304] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:27.758322 28050 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 20:00:27.806536 31279 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20250905 20:00:27.758778 28748 raft_consensus.cc:2391] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 5: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 20:00:27.727233 30946 raft_consensus.cc:695] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:27.812901 28083 raft_consensus.cc:1238] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20250905 20:00:27.757730 30777 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:27.802567 31203 raft_consensus.cc:3053] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 6 LEADER]: Stepping down as leader of term 6
W20250905 20:00:27.819356 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 756867us
I20250905 20:00:27.848513 28051 leader_election.cc:304] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 20:00:27.821972 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 55395.7us
W20250905 20:00:27.823014 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 221406us
I20250905 20:00:27.809675 31305 raft_consensus.cc:3053] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 LEADER]: Stepping down as leader of term 7
I20250905 20:00:27.860657 28123 leader_election.cc:304] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:27.817747 31282 raft_consensus.cc:3058] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:27.769219 31317 consensus_queue.cc:1035] T fd2b679c53994263846f51ceec97a150 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.059s
I20250905 20:00:27.843952 31187 leader_election.cc:290] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:27.867394 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 48807.7us
I20250905 20:00:27.757714 31325 raft_consensus.cc:513] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:27.841717 31250 consensus_meta.cc:287] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.856s user 0.008s sys 0.004s
W20250905 20:00:27.870368 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 259203us
I20250905 20:00:27.848788 30777 raft_consensus.cc:3058] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Advancing to term 9
I20250905 20:00:27.687731 31266 leader_election.cc:290] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:27.878001 28125 leader_election.cc:336] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 3.174s (SENT)
I20250905 20:00:27.848023 31203 raft_consensus.cc:738] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:27.881224 27973 leader_election.cc:304] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:27.882529 31190 raft_consensus.cc:2802] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Leader election won for term 9
I20250905 20:00:27.883152 31355 raft_consensus.cc:3053] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 LEADER]: Stepping down as leader of term 6
W20250905 20:00:27.887033 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 128786us
I20250905 20:00:27.887154 31221 raft_consensus.cc:2802] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:27.859395 31275 raft_consensus.cc:2747] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:27.901542 31266 raft_consensus.cc:2747] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250905 20:00:27.889047 28051 leader_election.cc:304] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:27.865005 31212 raft_consensus.cc:3053] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 8 LEADER]: Stepping down as leader of term 8
I20250905 20:00:27.913384 31212 raft_consensus.cc:738] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:27.914336 31212 consensus_queue.cc:260] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:27.860987 31305 raft_consensus.cc:738] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:27.915529 31356 raft_consensus.cc:2747] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:27.916267 31305 consensus_queue.cc:260] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:27.917304 31347 log.cc:927] Time spent T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.629s user 0.002s sys 0.000s
I20250905 20:00:27.917444 31305 raft_consensus.cc:3058] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:27.817135 31284 raft_consensus.cc:513] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:27.919232 28155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2322) took 1186 ms. Trace:
W20250905 20:00:27.881461 31252 consensus_meta.cc:287] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.623s user 0.011s sys 0.000s
I20250905 20:00:27.919634 28155 rpcz_store.cc:276] 0905 20:00:26.732731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:26.733013 (+ 282us) service_pool.cc:225] Handling call
0905 20:00:26.733513 (+ 500us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:26.911679 (+178166us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:27.918974 (+1007295us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b540009d8d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:27.919158 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":407778176}
I20250905 20:00:27.920122 31284 leader_election.cc:290] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:27.920265 31252 raft_consensus.cc:513] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:27.920836 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 20:00:27.921422 31228 consensus_meta.cc:287] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.942s user 0.010s sys 0.000s
I20250905 20:00:27.899516 27973 leader_election.cc:304] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:27.888295 31261 raft_consensus.cc:2747] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:27.907897 31084 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:27.908422 31196 raft_consensus.cc:2747] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:27.876029 30978 leader_election.cc:290] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:27.887902 30946 consensus_queue.cc:237] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:27.923154 31252 leader_election.cc:290] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:27.924203 31084 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 12 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:27.924633 27908 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:60818 (request call id 148) took 26225 ms (client timeout 15000 ms). Trace:
I20250905 20:00:27.925302 31084 raft_consensus.cc:2747] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:27.909559 31203 consensus_queue.cc:260] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:27.901255 31221 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:27.925801 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:27.926195 31221 raft_consensus.cc:3058] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:27.926337 31203 raft_consensus.cc:3058] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:27.914685 28051 leader_election.cc:304] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 20:00:27.931334 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:27.932478 28051 leader_election.cc:304] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:27.932725 31196 raft_consensus.cc:2747] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:27.933379 31196 consensus_queue.cc:1035] T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 20:00:27.934191 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:27.934911 31196 raft_consensus.cc:2747] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250905 20:00:27.934911 28743 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 11 FOLLOWER]: Advancing to term 12
I20250905 20:00:27.934198 28051 leader_election.cc:304] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 20:00:27.938609 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:27.939694 28051 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:27.940917 28051 leader_election.cc:304] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 20:00:27.941000 28153 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33752 (request call id 1733) took 2795 ms (client timeout 2007 ms). Trace:
I20250905 20:00:27.941744 31252 raft_consensus.cc:2747] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250905 20:00:27.941608 28153 rpcz_store.cc:269] 0905 20:00:25.145857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:25.201347 (+ 55490us) service_pool.cc:225] Handling call
0905 20:00:27.940732 (+2739385us) spinlock_profiling.cc:243] Waited 271 ms on lock 0x7b54002d0854. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:27.940897 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":622427008}
I20250905 20:00:27.942647 31252 raft_consensus.cc:2747] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250905 20:00:27.943471 31252 raft_consensus.cc:2747] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250905 20:00:27.898198 28125 leader_election.cc:336] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 3.364s (SENT)
W20250905 20:00:27.944490 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 45069.1us
W20250905 20:00:27.944602 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 46451.4us
I20250905 20:00:27.946658 27973 leader_election.cc:304] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
W20250905 20:00:27.947754 31345 log.cc:927] Time spent T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.473s user 0.002s sys 0.000s
I20250905 20:00:27.948565 27973 leader_election.cc:304] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:27.950395 31221 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:27.952061 28083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1682) took 2017 ms. Trace:
I20250905 20:00:27.952576 28083 rpcz_store.cc:276] 0905 20:00:25.934649 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:25.938268 (+ 3619us) service_pool.cc:225] Handling call
0905 20:00:26.927137 (+988869us) spinlock_profiling.cc:243] Waited 2.24 ms on lock 0x7b5800204f98. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b91f60 00007fe4a2b7d1a9 00007fe4a2b7cc52 00007fe49c202b44
0905 20:00:26.927366 (+ 229us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:27.828935 (+901569us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:27.951705 (+122770us) spinlock_profiling.cc:243] Waited 885 ms on lock 0x7b540023ad54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:27.951984 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2040856576}
I20250905 20:00:27.954882 31221 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:27.955343 28080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1684) took 2019 ms. Trace:
I20250905 20:00:27.955808 28080 rpcz_store.cc:276] 0905 20:00:25.935504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:26.473576 (+538072us) service_pool.cc:225] Handling call
0905 20:00:27.696982 (+1223406us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:27.697879 (+ 897us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:27.955113 (+257234us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b540019ebd0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:27.955274 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2812355840}
I20250905 20:00:27.956692 28051 leader_election.cc:304] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:27.958042 28750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40926 (request call id 551) took 1574 ms. Trace:
I20250905 20:00:27.958498 28750 rpcz_store.cc:276] 0905 20:00:26.443527 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:26.445855 (+ 2328us) service_pool.cc:225] Handling call
0905 20:00:26.446229 (+ 374us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:26.657108 (+210879us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:27.957781 (+1300673us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b54005302d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:27.957981 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":433625984}
I20250905 20:00:27.959007 31196 raft_consensus.cc:2747] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250905 20:00:27.959144 31348 log.cc:927] Time spent T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20250905 20:00:27.959460 28051 leader_election.cc:304] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:27.963869 31325 leader_election.cc:290] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:27.964972 28743 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 12.
I20250905 20:00:27.966130 28749 raft_consensus.cc:1238] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:27.970047 27973 leader_election.cc:304] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
W20250905 20:00:27.974684 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 343198us
I20250905 20:00:27.987636 28051 leader_election.cc:304] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:27.991819 28745 raft_consensus.cc:1238] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20250905 20:00:27.994539 28125 leader_election.cc:336] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.665s (SENT)
I20250905 20:00:27.995273 28125 leader_election.cc:304] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
W20250905 20:00:27.995996 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 95100.1us
I20250905 20:00:27.895835 31216 raft_consensus.cc:3058] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Advancing to term 7
W20250905 20:00:28.002494 28125 leader_election.cc:336] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.758s (SENT)
I20250905 20:00:27.980939 27904 catalog_manager.cc:5582] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 20:00:27.897558 31321 raft_consensus.cc:2802] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250905 20:00:28.005019 27904 catalog_manager.cc:5582] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250905 20:00:28.005734 28125 leader_election.cc:336] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.812s (SENT)
I20250905 20:00:28.011215 27975 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:28.011142 27974 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:27.897758 31355 raft_consensus.cc:738] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:27.903375 31190 raft_consensus.cc:695] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:27.893479 31250 raft_consensus.cc:513] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:28.011031 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:28.043080 28748 raft_consensus.cc:3058] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:27.974359 31084 raft_consensus.cc:2747] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:27.997473 31252 raft_consensus.cc:2802] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader election won for term 8
I20250905 20:00:28.012812 27973 leader_election.cc:304] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:28.055052 27973 leader_election.cc:304] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:28.056447 27973 leader_election.cc:304] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:28.057621 27973 leader_election.cc:304] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:28.058993 28751 raft_consensus.cc:2391] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 8: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 20:00:28.060294 31250 leader_election.cc:290] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:28.065686 31216 raft_consensus.cc:2747] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:27.959177 31228 consensus_queue.cc:1035] T a002b516fab044b998f136c341a88b74 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.081s
I20250905 20:00:28.066892 31309 raft_consensus.cc:2747] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250905 20:00:28.067457 31277 consensus_meta.cc:287] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.820s user 0.012s sys 0.000s
I20250905 20:00:28.067975 31277 raft_consensus.cc:2747] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250905 20:00:28.072748 28125 leader_election.cc:336] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421): Timed out: RequestConsensusVote RPC to 127.26.58.129:38421 timed out after 1.860s (SENT)
I20250905 20:00:28.079349 31252 raft_consensus.cc:695] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 8 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:28.080066 31279 raft_consensus.cc:2747] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250905 20:00:28.079798 31355 consensus_queue.cc:260] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.9, Last appended by leader: 9, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:28.080823 31279 raft_consensus.cc:2747] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:28.081180 31355 raft_consensus.cc:3058] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:28.012378 31321 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:28.081718 31321 raft_consensus.cc:3058] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Advancing to term 9
I20250905 20:00:28.041966 30978 raft_consensus.cc:2802] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 12 FOLLOWER]: Leader election won for term 12
I20250905 20:00:28.083405 30978 raft_consensus.cc:695] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 12 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:28.084411 31346 raft_consensus.cc:2802] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 20:00:28.084899 31346 raft_consensus.cc:491] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:28.085357 31346 raft_consensus.cc:3058] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:28.084587 30978 consensus_queue.cc:237] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:28.014057 31269 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:28.079517 31223 raft_consensus.cc:2747] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:28.086762 31269 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:28.051578 31266 raft_consensus.cc:2802] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:28.080611 31252 consensus_queue.cc:237] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:28.087749 31266 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:28.088320 31266 raft_consensus.cc:3058] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:28.089097 31269 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 20:00:28.052822 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 65277.4us
I20250905 20:00:28.017853 31284 raft_consensus.cc:2747] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:28.091217 28051 leader_election.cc:304] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:28.017576 31253 consensus_queue.cc:1035] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.086s
I20250905 20:00:28.092379 31220 raft_consensus.cc:2802] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election won for term 5
I20250905 20:00:28.093156 31220 raft_consensus.cc:695] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:27.996048 31299 consensus_queue.cc:1035] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.074s
I20250905 20:00:28.094091 31269 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:28.094596 31357 raft_consensus.cc:2747] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:28.094744 31269 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:28.095494 31227 leader_election.cc:290] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:28.018630 31210 leader_election.cc:290] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:28.094050 31220 consensus_queue.cc:237] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:28.098268 31306 raft_consensus.cc:2747] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250905 20:00:27.925045 27908 rpcz_store.cc:269] 0905 20:00:01.750168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:01.776543 (+ 26375us) service_pool.cc:225] Handling call
0905 20:00:27.924550 (+26148007us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 20:00:23.183016 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 20:00:23.183821 (+ 805us) write_op.cc:432] Acquiring schema lock in shared mode
0905 20:00:23.183876 (+ 55us) write_op.cc:435] Acquired schema lock
0905 20:00:23.183888 (+ 12us) tablet.cc:660] Decoding operations
0905 20:00:23.184680 (+ 792us) write_op.cc:620] Acquiring the partition lock for write op
0905 20:00:23.184823 (+ 143us) write_op.cc:641] Partition lock acquired for write op
0905 20:00:23.184865 (+ 42us) tablet.cc:683] Acquiring locks for 5 operations
0905 20:00:23.211461 (+ 26596us) tablet.cc:699] Row locks acquired
0905 20:00:23.211487 (+ 26us) write_op.cc:260] PREPARE: finished
0905 20:00:23.211781 (+ 294us) write_op.cc:270] Start()
0905 20:00:23.212011 (+ 230us) write_op.cc:276] Timestamp: P: 1757102423211677 usec, L: 0
0905 20:00:23.212036 (+ 25us) op_driver.cc:348] REPLICATION: starting
0905 20:00:23.212897 (+ 861us) log.cc:844] Serialized 2069 byte log entry
0905 20:00:23.507548 (+294651us) op_driver.cc:464] REPLICATION: finished
0905 20:00:23.693789 (+186241us) spinlock_profiling.cc:243] Waited 3.95 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49ac86fec 00007fe4a38dcb99 00007fe49aca369b 00007fe49aca193f 00007fe49aca3c02 00007fe4a1643e60
0905 20:00:23.861515 (+167726us) write_op.cc:301] APPLY: starting
0905 20:00:23.879880 (+ 18365us) tablet.cc:1365] starting BulkCheckPresence
0905 20:00:23.926182 (+ 46302us) tablet.cc:1368] finished BulkCheckPresence
0905 20:00:23.936780 (+ 10598us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 20:00:23.949809 (+ 13029us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 20:00:23.950175 (+ 366us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0905 20:00:23.950238 (+ 63us) write_op.cc:312] APPLY: finished
0905 20:00:23.950627 (+ 389us) log.cc:844] Serialized 47 byte log entry
0905 20:00:23.950843 (+ 216us) write_op.cc:489] Releasing partition, row and schema locks
0905 20:00:23.950990 (+ 147us) write_op.cc:454] Released schema lock
0905 20:00:23.951061 (+ 71us) write_op.cc:341] FINISH: Updating metrics
0905 20:00:23.999228 (+ 48167us) write_op.cc:489] Releasing partition, row and schema locks
0905 20:00:24.031039 (+ 31811us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":123752,"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":304010,"apply.run_cpu_time_us":3516,"apply.run_wall_time_us":128793,"num_ops":5,"prepare.queue_time_us":194911,"prepare.run_cpu_time_us":25676,"prepare.run_wall_time_us":80865,"raft.queue_time_us":155543,"raft.run_cpu_time_us":44848,"raft.run_wall_time_us":270549,"replication_time_us":295299,"spinlock_wait_cycles":9082624,"thread_start_us":549984,"threads_started":4,"wal-append.queue_time_us":60213,"wal-append.run_cpu_time_us":33726,"wal-append.run_wall_time_us":1880466}]]}
I20250905 20:00:28.095927 28051 leader_election.cc:400] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 election: Vote denied by peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421) with higher term. Message: Invalid argument: T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 7. Current term is 8.
I20250905 20:00:28.108168 28153 raft_consensus.cc:1238] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:28.108170 28051 leader_election.cc:403] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20250905 20:00:28.110682 28051 leader_election.cc:304] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:28.116771 31269 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:28.118583 31190 consensus_queue.cc:237] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:28.119015 31293 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:28.121141 31312 consensus_meta.cc:287] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 1.102s user 0.011s sys 0.000s
I20250905 20:00:28.133047 28748 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 6.
I20250905 20:00:28.133522 28122 leader_election.cc:304] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:28.138391 28122 leader_election.cc:304] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:28.140839 28715 leader_election.cc:304] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:28.148010 28122 leader_election.cc:304] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:28.149859 28122 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:28.012841 27904 catalog_manager.cc:5582] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 20:00:28.156966 28004 raft_consensus.cc:1273] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20250905 20:00:28.104665 31314 consensus_meta.cc:287] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.856s user 0.011s sys 0.000s
I20250905 20:00:28.184753 28122 leader_election.cc:304] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:28.010852 31221 consensus_queue.cc:1035] T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.115s
I20250905 20:00:28.052417 31317 raft_consensus.cc:491] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:28.156787 27904 catalog_manager.cc:5582] T 2bd6bb972f14472a89ccf99de880efc2 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 1 to 6, leader changed from cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) to 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132). New cstate: current_term: 6 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 20:00:28.195974 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:28.199195 28008 raft_consensus.cc:3058] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
W20250905 20:00:28.139937 31177 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.851s user 0.004s sys 0.009s
I20250905 20:00:28.204988 27904 catalog_manager.cc:5582] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 20:00:28.207818 27973 leader_election.cc:304] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:28.207788 27904 catalog_manager.cc:5582] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 20:00:28.210300 28007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
W20250905 20:00:28.211323 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:28.213157 31084 consensus_queue.cc:1035] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250905 20:00:28.213681 31321 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:28.214802 30775 consensus_queue.cc:1035] T 5c21c3755f90405db03f98780efdb0b5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 20:00:28.213207 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:28.216797 31321 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 9 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:28.217417 28080 raft_consensus.cc:2466] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 4.
I20250905 20:00:28.218974 28083 raft_consensus.cc:1238] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:28.125296 31312 raft_consensus.cc:513] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:28.151173 31269 raft_consensus.cc:2747] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250905 20:00:28.220943 31269 raft_consensus.cc:2802] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250905 20:00:28.221331 31269 raft_consensus.cc:491] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:28.221693 31269 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Advancing to term 10
I20250905 20:00:28.221964 31312 leader_election.cc:290] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:28.134812 31228 raft_consensus.cc:3058] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:28.223237 28002 raft_consensus.cc:2391] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 5: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:28.223958 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:28.224668 28003 raft_consensus.cc:3058] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:28.226573 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a042aed9ffcb49c19e07c53c69bd1b95" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:28.227420 28151 raft_consensus.cc:3058] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:28.231789 31084 raft_consensus.cc:2747] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:28.231889 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:28.231442 31317 raft_consensus.cc:513] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:28.135442 31332 raft_consensus.cc:2747] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250905 20:00:28.243300 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:28.242297 28008 raft_consensus.cc:2466] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 6.
W20250905 20:00:28.247282 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:28.257225 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:28.258565 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:28.259518 28008 raft_consensus.cc:2366] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 10. Current term is 11.
I20250905 20:00:28.260388 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:28.261832 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 20:00:28.195487 31347 log.cc:927] Time spent T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.126s user 0.000s sys 0.001s
W20250905 20:00:28.274653 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:28.197484 31315 raft_consensus.cc:2802] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 20:00:28.275884 28085 raft_consensus.cc:1238] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20250905 20:00:28.267622 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:28.135085 31284 consensus_queue.cc:1046] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.121s
I20250905 20:00:28.209370 31227 consensus_queue.cc:1046] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20250905 20:00:28.287292 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 20:00:28.238003 31177 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.954s user 0.005s sys 0.010s
I20250905 20:00:28.289733 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:28.192406 31277 consensus_queue.cc:1046] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.083s
I20250905 20:00:28.292586 28077 raft_consensus.cc:1273] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20250905 20:00:28.296610 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:28.382668 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:28.205003 31314 raft_consensus.cc:513] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:28.299340 28156 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.017s sys 0.012s
W20250905 20:00:28.383635 28156 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.658s user 0.018s sys 0.012s
W20250905 20:00:28.300657 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:28.384927 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:28.305927 27904 catalog_manager.cc:5582] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 20:00:28.307737 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:28.385953 28084 raft_consensus.cc:3058] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:28.387801 27904 catalog_manager.cc:5582] T 62ccdb07048e4762b1b31c36b14610f5 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 5 to 6, leader changed from cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) to 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132). New cstate: current_term: 6 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 20:00:28.342988 28151 raft_consensus.cc:2466] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 6.
W20250905 20:00:28.348453 28079 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.774s user 0.006s sys 0.011s
I20250905 20:00:28.349750 28749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40918 (request call id 655) took 1638 ms. Trace:
I20250905 20:00:28.351138 28008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:28.352563 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:28.397684 28749 rpcz_store.cc:276] 0905 20:00:26.711812 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:26.711960 (+ 148us) service_pool.cc:225] Handling call
0905 20:00:27.965952 (+1253992us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:27.966589 (+ 637us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:28.349491 (+382902us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54005302d0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:28.349677 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2832359808}
I20250905 20:00:28.399461 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:28.355479 31275 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:28.366720 28002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 2190) took 1399 ms. Trace:
I20250905 20:00:28.410812 28002 rpcz_store.cc:276] 0905 20:00:26.967455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:27.017760 (+ 50305us) service_pool.cc:225] Handling call
0905 20:00:28.359047 (+1341287us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540034c454. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:28.366631 (+ 7584us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2555706112}
I20250905 20:00:28.412416 28006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 2172) took 2969 ms. Trace:
W20250905 20:00:28.367628 31353 log.cc:927] Time spent T 1b18b7149ff3439a8dde09149cf336b3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.494s user 0.009s sys 0.000s
I20250905 20:00:28.370452 28743 raft_consensus.cc:1273] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250905 20:00:28.412824 28006 rpcz_store.cc:276] 0905 20:00:25.442834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:25.648957 (+206123us) service_pool.cc:225] Handling call
0905 20:00:27.030772 (+1381815us) spinlock_profiling.cc:243] Waited 3.33 ms on lock 0x7b5800000f98. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b91f60 00007fe4a2b7d1a9 00007fe4a2b7cc52 00007fe49c202b44
0905 20:00:27.031013 (+ 241us) raft_consensus.cc:1469] Updating replica for 2 ops
0905 20:00:27.031410 (+ 397us) raft_consensus.cc:1532] Early marking committed up to index 2
0905 20:00:27.031749 (+ 339us) log.cc:844] Serialized 14 byte log entry
0905 20:00:27.288253 (+256504us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0905 20:00:27.290037 (+ 1784us) log.cc:844] Serialized 52 byte log entry
0905 20:00:27.290282 (+ 245us) raft_consensus.cc:1643] Marking committed up to 3
0905 20:00:27.357236 (+ 66954us) log.cc:844] Serialized 14 byte log entry
0905 20:00:27.357497 (+ 261us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:27.357577 (+ 80us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:27.916682 (+559105us) spinlock_profiling.cc:243] Waited 4.96 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49d85891e 00007fe49d857c66 00007fe49d857888 00007fe49d8612c1 00007fe49d861279 00007fe49d861209
0905 20:00:28.412132 (+495450us) raft_consensus.cc:1683] finished
0905 20:00:28.412162 (+ 30us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:28.412353 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":42697,"prepare.queue_time_us":66033,"prepare.run_cpu_time_us":116,"prepare.run_wall_time_us":2026,"raft.queue_time_us":1400,"raft.run_cpu_time_us":461,"raft.run_wall_time_us":11888,"spinlock_wait_cycles":19050624,"thread_start_us":279602,"threads_started":2,"wal-append.queue_time_us":256397}
W20250905 20:00:28.371552 28159 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.521s user 0.009s sys 0.011s
I20250905 20:00:28.297998 31317 leader_election.cc:290] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 20:00:28.380324 28752 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.011s sys 0.011s
I20250905 20:00:28.391435 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:28.394222 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:28.420938 28083 raft_consensus.cc:2466] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 8.
W20250905 20:00:28.347749 31213 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.648s user 0.000s sys 0.027s
I20250905 20:00:28.390260 27904 catalog_manager.cc:5582] T b13af7609cd04cf5b21745527b70074d P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 2 to 7. New cstate: current_term: 7 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250905 20:00:28.354583 31360 log.cc:927] Time spent T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.263s user 0.003s sys 0.000s
I20250905 20:00:28.347574 31315 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
W20250905 20:00:28.400907 28079 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.828s user 0.006s sys 0.011s
I20250905 20:00:28.425803 28079 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59564 (request call id 1685) took 2489 ms. Trace:
I20250905 20:00:28.401062 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:28.428406 28153 raft_consensus.cc:3058] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
W20250905 20:00:28.404748 28150 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.031s sys 0.023s
W20250905 20:00:28.429405 28150 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.537s user 0.031s sys 0.024s
I20250905 20:00:28.431391 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:28.433540 28715 leader_election.cc:304] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 20:00:28.402408 31282 consensus_meta.cc:287] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.513s user 0.011s sys 0.000s
I20250905 20:00:28.418416 28050 leader_election.cc:304] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:28.440341 27904 catalog_manager.cc:5582] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 3 to 6, leader changed from 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) to e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130). New cstate: current_term: 6 leader_uuid: "e4413ae5e0b64b52a6f360ac580239b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 20:00:28.383850 31177 raft_consensus.cc:695] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:28.426273 28079 rpcz_store.cc:276] 0905 20:00:25.935897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:26.483456 (+547559us) service_pool.cc:225] Handling call
0905 20:00:26.484006 (+ 550us) raft_consensus.cc:1469] Updating replica for 2 ops
0905 20:00:26.501433 (+ 17427us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:26.501486 (+ 53us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0905 20:00:26.507966 (+ 6480us) log.cc:844] Serialized 52 byte log entry
0905 20:00:26.940289 (+432323us) spinlock_profiling.cc:243] Waited 19.2 ms on lock 0x7b180010d920. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49ac86d47 00007fe4a38dcb99 00007fe49aca369b 00007fe49aca193f 00007fe49aca3c02 00007fe49d8572bb 00007fe49d85cdc3 00007fe49d85d171 00007fe49dbec09f
0905 20:00:27.034765 (+ 94476us) raft_consensus.cc:1643] Marking committed up to 3
0905 20:00:27.074297 (+ 39532us) log.cc:844] Serialized 14 byte log entry
0905 20:00:27.569523 (+495226us) spinlock_profiling.cc:243] Waited 5.22 ms on lock 0x7b180010d920. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49ac19f4a 00007fe4a4107409 00007fe49aadb3e0 00007fe49aca14e4 00007fe49aca1455 00007fe49dc11020 00007fe49dc236c6 00007fe49dc34237 00007fe49dc341b8
0905 20:00:28.425300 (+855777us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:28.425410 (+ 110us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:28.425497 (+ 87us) raft_consensus.cc:1683] finished
0905 20:00:28.425519 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:28.425720 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19810,"prepare.queue_time_us":39265,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":15672,"raft.queue_time_us":530555,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":18665,"spinlock_wait_cycles":56216704,"thread_start_us":949626,"threads_started":2,"wal-append.queue_time_us":682254}
I20250905 20:00:28.477170 28084 raft_consensus.cc:2466] T 60054c57e2554f32a5ad379a79c2a738 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 8.
W20250905 20:00:28.421806 28752 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.574s user 0.011s sys 0.011s
I20250905 20:00:28.437947 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:28.439206 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:28.481092 28156 raft_consensus.cc:2466] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
W20250905 20:00:28.412210 31331 log.cc:927] Time spent T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.069s user 0.001s sys 0.000s
W20250905 20:00:28.485498 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 52080.6us
I20250905 20:00:28.343679 30777 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:28.422883 31213 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.726s user 0.000s sys 0.028s
I20250905 20:00:28.446579 31210 raft_consensus.cc:2802] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 20:00:28.448012 31221 raft_consensus.cc:2802] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election won for term 6
I20250905 20:00:28.458315 31266 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:28.401108 31299 raft_consensus.cc:2762] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20250905 20:00:28.433053 31313 raft_consensus.cc:491] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
W20250905 20:00:28.501967 31367 log.cc:927] Time spent T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
I20250905 20:00:28.502276 31313 raft_consensus.cc:513] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:28.458315 31228 raft_consensus.cc:2747] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421) with higher term. Message: Invalid argument: T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for earlier term 7. Current term is 8.
I20250905 20:00:28.458581 31358 raft_consensus.cc:491] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:28.503674 31358 raft_consensus.cc:513] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:28.505771 31358 leader_election.cc:290] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:28.460121 31314 leader_election.cc:290] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:28.391816 31325 raft_consensus.cc:2747] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:28.423659 31275 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:28.507963 31341 raft_consensus.cc:2802] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 20:00:28.508531 31275 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:28.508602 31341 raft_consensus.cc:491] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:28.509349 31341 raft_consensus.cc:3058] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:28.510886 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:28.486971 31282 raft_consensus.cc:513] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:28.511871 28749 raft_consensus.cc:2391] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 9: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 20:00:28.513025 28080 raft_consensus.cc:2466] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 8.
I20250905 20:00:28.513266 31282 leader_election.cc:290] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:28.513983 28009 raft_consensus.cc:2391] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 6: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:28.514053 31227 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:28.464112 31322 consensus_queue.cc:1046] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20250905 20:00:28.419232 31357 raft_consensus.cc:3053] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 4 LEADER]: Stepping down as leader of term 4
W20250905 20:00:28.465162 31151 consensus_meta.cc:287] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 1.508s user 0.012s sys 0.000s
I20250905 20:00:28.515844 31357 raft_consensus.cc:738] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:28.516058 31151 raft_consensus.cc:513] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:28.465478 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:28.466325 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:28.519623 28002 raft_consensus.cc:2391] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 6: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
W20250905 20:00:28.520601 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 112564us
I20250905 20:00:28.463624 28003 raft_consensus.cc:2466] T a042aed9ffcb49c19e07c53c69bd1b95 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 6.
I20250905 20:00:28.481127 28078 raft_consensus.cc:2466] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 10.
I20250905 20:00:28.482645 27975 leader_election.cc:304] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:28.526153 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:28.527294 28003 raft_consensus.cc:2466] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
I20250905 20:00:28.489197 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:28.403355 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:28.461798 31346 raft_consensus.cc:513] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:28.439301 31306 raft_consensus.cc:2802] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:28.433970 31315 raft_consensus.cc:3058] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:28.543382 28157 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:28.514295 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:28.514168 31221 raft_consensus.cc:695] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:28.515111 31227 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:28.379858 31171 consensus_meta.cc:287] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.713s user 0.021s sys 0.001s
W20250905 20:00:28.420433 28159 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.570s user 0.009s sys 0.011s
I20250905 20:00:28.529610 28151 raft_consensus.cc:3058] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Advancing to term 9
I20250905 20:00:28.482228 27904 catalog_manager.cc:5582] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 4 to 7. New cstate: current_term: 7 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 20:00:28.530259 28153 raft_consensus.cc:2466] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 20:00:28.560087 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:28.550024 28051 leader_election.cc:304] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:28.562299 28051 leader_election.cc:304] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:28.555881 27975 leader_election.cc:304] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:28.560221 28717 leader_election.cc:304] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:28.563418 27904 catalog_manager.cc:5582] T 9f7f6e4e3a754dcda90928621168fbe7 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 1 to 6, leader changed from cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) to 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132). New cstate: current_term: 6 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 20:00:28.548522 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:28.565750 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:28.566130 28156 raft_consensus.cc:3058] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:28.569547 28159 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1753) took 1032 ms. Trace:
I20250905 20:00:28.571580 28007 raft_consensus.cc:2391] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 8: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:28.568542 27904 catalog_manager.cc:5582] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: term changed from 3 to 4, leader changed from 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132) to cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129). New cstate: current_term: 4 leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 20:00:28.569924 28159 rpcz_store.cc:276] 0905 20:00:27.537450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:27.676358 (+138908us) service_pool.cc:225] Handling call
0905 20:00:27.678931 (+ 2573us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:27.679377 (+ 446us) raft_consensus.cc:1532] Early marking committed up to index 4
0905 20:00:27.850026 (+170649us) log.cc:844] Serialized 14 byte log entry
0905 20:00:28.569193 (+719167us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:28.569245 (+ 52us) raft_consensus.cc:1643] Marking committed up to 4
0905 20:00:28.569297 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:28.569369 (+ 72us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:28.569492 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9836,"prepare.queue_time_us":218456,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":17028,"thread_start_us":874113,"threads_started":2}
I20250905 20:00:28.573837 28082 raft_consensus.cc:2391] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 9: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:28.574527 28003 raft_consensus.cc:2408] T d06337b10d154efbbaaf3c152d01f6dd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250905 20:00:28.545861 31284 raft_consensus.cc:2802] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:28.572395 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:28.514045 31216 consensus_queue.cc:1035] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
W20250905 20:00:28.573500 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 380572us
I20250905 20:00:28.580646 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:28.578356 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:28.592449 28122 leader_election.cc:304] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:28.593140 28159 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 10 FOLLOWER]: Advancing to term 11
I20250905 20:00:28.587697 28155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1756) took 1032 ms. Trace:
I20250905 20:00:28.587877 28154 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1751) took 1074 ms. Trace:
I20250905 20:00:28.584694 28150 raft_consensus.cc:1238] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:28.560317 31293 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 11 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:28.593649 28155 rpcz_store.cc:276] 0905 20:00:27.555268 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:27.921720 (+366452us) service_pool.cc:225] Handling call
0905 20:00:27.922007 (+ 287us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:27.922221 (+ 214us) raft_consensus.cc:1532] Early marking committed up to index 5
0905 20:00:27.978652 (+ 56431us) log.cc:844] Serialized 14 byte log entry
0905 20:00:28.587202 (+608550us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:28.587283 (+ 81us) raft_consensus.cc:1643] Marking committed up to 5
0905 20:00:28.587407 (+ 124us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:28.587527 (+ 120us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:28.587638 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":167919,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":49087,"raft.run_cpu_time_us":68,"raft.run_wall_time_us":68,"thread_start_us":607886,"threads_started":2,"wal-append.queue_time_us":594070}
I20250905 20:00:28.749331 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:28.570611 31196 raft_consensus.cc:2802] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election won for term 6
W20250905 20:00:28.702297 31220 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.598s user 0.000s sys 0.033s
I20250905 20:00:28.676393 31177 consensus_queue.cc:237] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:28.577839 31213 raft_consensus.cc:3058] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:28.660571 31314 raft_consensus.cc:2747] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:28.668300 28151 raft_consensus.cc:2466] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 9.
I20250905 20:00:28.779239 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:28.781424 27975 leader_election.cc:304] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 20:00:28.762051 31220 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.660s user 0.005s sys 0.033s
I20250905 20:00:28.676245 31228 raft_consensus.cc:491] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:28.621011 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:28.559114 31275 consensus_queue.cc:1035] T 8e0ac2dfe59d423189dd8d72eee98a42 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.063s
I20250905 20:00:28.516777 31357 consensus_queue.cc:260] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:28.640966 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:28.590955 31151 leader_election.cc:290] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:28.727051 31379 log.cc:927] Time spent T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.090s user 0.001s sys 0.000s
W20250905 20:00:28.736799 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:28.648600 31221 consensus_queue.cc:237] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:28.671429 28159 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 11.
I20250905 20:00:28.570595 31171 raft_consensus.cc:513] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:28.583868 27904 catalog_manager.cc:5582] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a reported cstate change: config changed from index 4 to 6, term changed from 4 to 7, leader changed from 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131) to 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132), VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New cstate: current_term: 7 leader_uuid: "7e9c9f591f05464f9b097bf003d14f4a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 20:00:28.768808 31266 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:28.617893 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:28.566414 31309 raft_consensus.cc:2747] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:28.631929 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:28.833087 28078 raft_consensus.cc:2466] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 7.
I20250905 20:00:28.584407 31284 raft_consensus.cc:491] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:28.623447 28154 rpcz_store.cc:276] 0905 20:00:27.513950 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:27.675739 (+161789us) service_pool.cc:225] Handling call
0905 20:00:27.676163 (+ 424us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:27.677062 (+ 899us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:27.677118 (+ 56us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:27.731326 (+ 54208us) log.cc:844] Serialized 26 byte log entry
0905 20:00:28.199473 (+468147us) raft_consensus.cc:1643] Marking committed up to 9
0905 20:00:28.259189 (+ 59716us) log.cc:844] Serialized 14 byte log entry
0905 20:00:28.584703 (+325514us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:28.585329 (+ 626us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:28.585553 (+ 224us) raft_consensus.cc:1683] finished
0905 20:00:28.585757 (+ 204us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:28.587638 (+ 1881us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":63256,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":50,"raft.queue_time_us":98173,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":34552,"thread_start_us":701308,"threads_started":2,"wal-append.queue_time_us":640145}
I20250905 20:00:28.752310 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:28.753474 28150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1759) took 1197 ms. Trace:
I20250905 20:00:28.838554 28150 rpcz_store.cc:276] 0905 20:00:27.555763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:28.560013 (+1004250us) service_pool.cc:225] Handling call
0905 20:00:28.584504 (+ 24491us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:28.752947 (+168443us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:28.753392 (+ 445us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 20:00:28.737075 31252 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.645s user 0.018s sys 0.010s
I20250905 20:00:28.763813 28081 raft_consensus.cc:2953] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 4 to 6, VOTER cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } } }
I20250905 20:00:28.711488 31391 consensus_queue.cc:1046] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20250905 20:00:28.737666 30777 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 9 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:28.789201 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 157039us
I20250905 20:00:28.646765 31190 raft_consensus.cc:491] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:28.747499 31346 leader_election.cc:290] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:28.800360 28002 raft_consensus.cc:2391] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 9: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:28.814268 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:28.822402 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:28.768466 31227 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:28.772106 31196 raft_consensus.cc:695] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:28.832093 28006 raft_consensus.cc:2366] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 6. Current term is 7.
W20250905 20:00:28.793534 31269 consensus_meta.cc:287] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.571s user 0.010s sys 0.006s
I20250905 20:00:28.835752 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:28.836879 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:28.844082 31284 raft_consensus.cc:3058] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:28.850935 28154 raft_consensus.cc:1238] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 7. Current term is 8. Ops: []
W20250905 20:00:28.835546 31369 log.cc:927] Time spent T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.200s user 0.000s sys 0.001s
I20250905 20:00:28.868657 28122 leader_election.cc:304] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:28.911588 31314 raft_consensus.cc:491] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:28.912269 31314 raft_consensus.cc:513] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:28.917202 31261 raft_consensus.cc:2747] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250905 20:00:28.880337 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:28.921242 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:28.922083 28006 raft_consensus.cc:2466] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
I20250905 20:00:28.805684 31228 raft_consensus.cc:513] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:28.924494 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:28.797492 31383 raft_consensus.cc:491] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:28.944617 31383 raft_consensus.cc:513] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:28.850625 31357 raft_consensus.cc:3058] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:28.892560 28154 raft_consensus.cc:1273] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250905 20:00:28.956959 28154 raft_consensus.cc:1238] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20250905 20:00:28.894508 31352 log.cc:927] Time spent T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 1.021s user 0.000s sys 0.008s
W20250905 20:00:28.895381 31385 log.cc:927] Time spent T 456d9fe0918b423abeab5c39372bcbe9 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.143s user 0.000s sys 0.001s
W20250905 20:00:28.896155 31374 log.cc:927] Time spent T 6dbd54765afc49cb862b267d6663000c P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.193s user 0.000s sys 0.001s
I20250905 20:00:28.887686 31190 raft_consensus.cc:513] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:28.899060 31376 log.cc:927] Time spent T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.102s user 0.000s sys 0.001s
W20250905 20:00:28.899721 31372 log.cc:927] Time spent T d06337b10d154efbbaaf3c152d01f6dd P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.165s user 0.001s sys 0.000s
I20250905 20:00:28.884852 28079 raft_consensus.cc:2466] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 6.
W20250905 20:00:28.901129 31371 log.cc:927] Time spent T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.131s user 0.000s sys 0.000s
I20250905 20:00:28.901165 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:28.902977 31389 log.cc:927] Time spent T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.155s user 0.000s sys 0.000s
W20250905 20:00:28.848304 31252 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.757s user 0.018s sys 0.010s
I20250905 20:00:28.926204 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:28.926630 28156 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 6.
I20250905 20:00:28.928643 28010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 2206) took 1891 ms. Trace:
I20250905 20:00:28.929083 31196 consensus_queue.cc:237] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:28.774564 31388 consensus_queue.cc:1046] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.121s
I20250905 20:00:28.920123 31391 raft_consensus.cc:3053] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 LEADER]: Stepping down as leader of term 7
W20250905 20:00:28.930460 31368 log.cc:927] Time spent T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.252s user 0.001s sys 0.001s
I20250905 20:00:28.969740 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:28.970606 28002 raft_consensus.cc:2466] T 6db33851e9914ca58560e8902701dca3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 10.
I20250905 20:00:28.888430 31084 raft_consensus.cc:2802] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Leader election won for term 9
I20250905 20:00:28.919734 31269 raft_consensus.cc:513] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:28.919775 31373 log.cc:927] Time spent T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.241s user 0.000s sys 0.001s
I20250905 20:00:28.936677 31171 leader_election.cc:290] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:28.937458 31210 raft_consensus.cc:491] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:28.941998 31321 raft_consensus.cc:3053] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Stepping down as leader of term 4
I20250905 20:00:28.947655 31383 leader_election.cc:290] T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:28.951889 28003 raft_consensus.cc:2391] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 10: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 20:00:29.017038 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:29.018218 28003 raft_consensus.cc:2408] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20250905 20:00:28.962224 28010 rpcz_store.cc:276] 0905 20:00:27.036871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:27.286093 (+249222us) service_pool.cc:225] Handling call
0905 20:00:27.298961 (+ 12868us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:27.299433 (+ 472us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:27.299467 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:27.299984 (+ 517us) log.cc:844] Serialized 26 byte log entry
0905 20:00:27.359049 (+ 59065us) raft_consensus.cc:1643] Marking committed up to 3
0905 20:00:27.359097 (+ 48us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:27.359170 (+ 73us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:28.927723 (+1568553us) spinlock_profiling.cc:243] Waited 33.9 ms on lock 0x7b24001a60d8. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49aaf9784 00007fe49aaf788b 00007fe49ab86669 00007fe49ab862f5 00007fe49ab85850 00007fe49d885498 00007fe49d885bf8 00007fe49d85d438 00007fe49d85881a 00007fe49d857c66
0905 20:00:28.928310 (+ 587us) raft_consensus.cc:1683] finished
0905 20:00:28.928337 (+ 27us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:28.928578 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":121,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":85,"spinlock_wait_cycles":78078720,"thread_start_us":58683,"threads_started":1,"wal-append.queue_time_us":315927}
I20250905 20:00:28.932582 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:28.948920 31328 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.018s sys 0.053s
I20250905 20:00:28.944638 31228 leader_election.cc:290] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:28.977516 28125 leader_election.cc:304] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:28.984737 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:28.972493 31388 raft_consensus.cc:3053] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Stepping down as leader of term 4
I20250905 20:00:28.983817 31084 raft_consensus.cc:695] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:28.990979 31269 leader_election.cc:290] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 10 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:28.994621 28157 raft_consensus.cc:1238] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 10 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 9. Current term is 10. Ops: []
I20250905 20:00:28.987943 31321 raft_consensus.cc:738] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:28.968022 31391 raft_consensus.cc:738] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
W20250905 20:00:29.055665 31328 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.666s user 0.018s sys 0.053s
I20250905 20:00:28.987669 31210 raft_consensus.cc:3058] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:29.007359 28153 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2333) took 1420 ms. Trace:
I20250905 20:00:29.008491 28158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1754) took 1458 ms. Trace:
I20250905 20:00:28.966463 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:28.967780 31314 leader_election.cc:290] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 20:00:29.019161 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:29.061733 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 84326.8us
I20250905 20:00:29.062161 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:29.075017 28158 rpcz_store.cc:276] 0905 20:00:27.550284 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:27.825498 (+275214us) service_pool.cc:225] Handling call
0905 20:00:27.826640 (+ 1142us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:27.878360 (+ 51720us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:27.878414 (+ 54us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:27.894264 (+ 15850us) log.cc:844] Serialized 26 byte log entry
0905 20:00:28.396752 (+502488us) raft_consensus.cc:1643] Marking committed up to 4
0905 20:00:28.396817 (+ 65us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:28.396933 (+ 116us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:29.008130 (+611197us) raft_consensus.cc:1683] finished
0905 20:00:29.008170 (+ 40us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:29.008405 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39938,"raft.queue_time_us":45583,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":7259,"thread_start_us":481855,"threads_started":1,"wal-append.queue_time_us":663462}
I20250905 20:00:29.063542 28001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 2207) took 2026 ms. Trace:
I20250905 20:00:29.066644 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:29.077687 26858 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 20:00:29.070017 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:29.078688 28009 raft_consensus.cc:2466] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:29.063493 28002 raft_consensus.cc:2466] T b13af7609cd04cf5b21745527b70074d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
I20250905 20:00:29.077476 28001 rpcz_store.cc:276] 0905 20:00:27.037173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:27.312500 (+275327us) service_pool.cc:225] Handling call
0905 20:00:27.313025 (+ 525us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:27.313522 (+ 497us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:27.313566 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:27.314372 (+ 806us) log.cc:844] Serialized 26 byte log entry
0905 20:00:27.546990 (+232618us) raft_consensus.cc:1643] Marking committed up to 4
0905 20:00:27.547167 (+ 177us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:27.547289 (+ 122us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:28.764973 (+1217684us) spinlock_profiling.cc:243] Waited 2.31 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49d85891e 00007fe49d857c66 00007fe49d857888 00007fe49d8612c1 00007fe49d861279 00007fe49d861209
0905 20:00:29.056694 (+291721us) raft_consensus.cc:1683] finished
0905 20:00:29.056729 (+ 35us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:29.063445 (+ 6716us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":145,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":102,"spinlock_wait_cycles":5317888,"thread_start_us":186401,"threads_started":1,"wal-append.queue_time_us":288344}
I20250905 20:00:29.073835 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:29.077008 28004 raft_consensus.cc:2408] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20250905 20:00:29.004325 31190 leader_election.cc:290] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:29.074772 28153 rpcz_store.cc:276] 0905 20:00:27.587250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:28.561745 (+974495us) service_pool.cc:225] Handling call
0905 20:00:28.562219 (+ 474us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:28.573347 (+ 11128us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:28.573389 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:28.582079 (+ 8690us) log.cc:844] Serialized 26 byte log entry
0905 20:00:28.827123 (+245044us) raft_consensus.cc:1643] Marking committed up to 4
0905 20:00:28.827186 (+ 63us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:28.827300 (+ 114us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:29.006993 (+179693us) raft_consensus.cc:1683] finished
0905 20:00:29.007026 (+ 33us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:29.007264 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5699,"raft.queue_time_us":28255,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":11824,"thread_start_us":184987,"threads_started":1,"wal-append.queue_time_us":385905}
I20250905 20:00:29.086920 28153 raft_consensus.cc:1238] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 8. Current term is 9. Ops: []
I20250905 20:00:29.097064 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:29.098371 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:29.099419 28004 raft_consensus.cc:2391] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 8: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:29.097240 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:29.063124 31388 raft_consensus.cc:738] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:29.069458 31383 raft_consensus.cc:2802] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20250905 20:00:29.095857 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 314479us
I20250905 20:00:29.101605 28125 leader_election.cc:304] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:29.107496 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:29.103209 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:29.109292 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:29.110404 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:29.110721 28125 leader_election.cc:304] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:29.111652 28003 raft_consensus.cc:2408] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20250905 20:00:29.109661 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:29.113771 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:29.115398 28010 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60054c57e2554f32a5ad379a79c2a738" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:29.120110 28010 raft_consensus.cc:3058] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:29.116410 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
W20250905 20:00:29.117228 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 248072us
I20250905 20:00:29.121663 28006 raft_consensus.cc:2391] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 8: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
W20250905 20:00:29.117905 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 249339us
I20250905 20:00:29.122285 28004 raft_consensus.cc:2466] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 8.
I20250905 20:00:29.124826 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:29.122318 27973 leader_election.cc:304] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:29.128104 28125 leader_election.cc:304] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:29.128247 27973 leader_election.cc:304] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:29.131037 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:29.135249 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:29.132516 27975 leader_election.cc:304] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:29.136994 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:29.138132 28004 raft_consensus.cc:2408] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20250905 20:00:29.141144 28009 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:29.141610 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:29.141885 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:29.142081 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:29.138582 31263 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.670s user 0.012s sys 0.006s
I20250905 20:00:29.142791 28080 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 10.
I20250905 20:00:29.146283 28002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
I20250905 20:00:29.107371 31383 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:29.154059 28125 leader_election.cc:304] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:29.149717 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:29.150113 28076 raft_consensus.cc:1273] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250905 20:00:29.150552 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:29.157104 28003 raft_consensus.cc:2408] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20250905 20:00:29.157320 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:29.152742 28079 raft_consensus.cc:1273] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250905 20:00:29.126667 31190 raft_consensus.cc:2747] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250905 20:00:29.155887 28006 raft_consensus.cc:2466] T b8a1b71a731e455da7990629d2c94801 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:29.132966 31309 raft_consensus.cc:2747] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:29.129853 31269 raft_consensus.cc:2747] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250905 20:00:29.164587 28125 leader_election.cc:304] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:29.165465 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:29.165944 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:29.203383 28079 raft_consensus.cc:2391] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 8: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 20:00:29.203267 28076 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 11.
I20250905 20:00:29.168257 28154 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:33742 (request call id 728) took 1545 ms. Trace:
I20250905 20:00:29.170115 28122 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:29.204531 28154 rpcz_store.cc:276] 0905 20:00:27.623296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:29.166750 (+1543454us) service_pool.cc:225] Handling call
0905 20:00:29.168167 (+ 1417us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:29.239442 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:29.240643 28080 raft_consensus.cc:2391] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 6: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:29.246660 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:29.144369 31328 raft_consensus.cc:2802] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:29.247502 28076 raft_consensus.cc:3058] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 11 FOLLOWER]: Advancing to term 12
W20250905 20:00:29.244081 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 79568.7us
W20250905 20:00:29.143684 31263 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.676s user 0.012s sys 0.006s
I20250905 20:00:29.153683 31171 raft_consensus.cc:2802] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 11 FOLLOWER]: Leader election won for term 11
I20250905 20:00:29.275117 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e802ce1da002477b8896acf7bf7c84a7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:29.274454 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:29.160848 31383 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 10 FOLLOWER]: Advancing to term 11
I20250905 20:00:29.096508 31321 consensus_queue.cc:260] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:29.270562 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:29.278587 28004 raft_consensus.cc:1238] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250905 20:00:29.163580 31084 consensus_queue.cc:237] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:29.148370 31391 consensus_queue.cc:260] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:29.523699 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:29.357190 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 187192us
W20250905 20:00:29.359530 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:29.378643 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 818427us
W20250905 20:00:29.530936 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:29.531471 28717 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:29.531559 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:29.244837 30946 consensus_queue.cc:1035] T 6c423052d6454829bccfe7c8bc1bda5c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.288s
I20250905 20:00:29.357383 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:29.387686 28076 raft_consensus.cc:2466] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 12.
W20250905 20:00:29.378885 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 591472us
W20250905 20:00:29.378364 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 863042us
I20250905 20:00:29.299387 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:29.374748 28752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40904 (request call id 780) took 1528 ms. Trace:
W20250905 20:00:29.271240 31341 consensus_meta.cc:287] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 0.739s user 0.017s sys 0.000s
W20250905 20:00:29.279513 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 102400us
I20250905 20:00:29.538282 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:29.544044 28083 raft_consensus.cc:3058] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Advancing to term 11
I20250905 20:00:29.171437 31261 raft_consensus.cc:2747] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250905 20:00:29.163470 31398 raft_consensus.cc:2802] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250905 20:00:29.534945 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:29.528668 28122 leader_election.cc:304] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:29.540380 28004 raft_consensus.cc:1238] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20250905 20:00:29.243992 31397 raft_consensus.cc:491] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
W20250905 20:00:29.378973 31359 log.cc:927] Time spent T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.132s user 0.000s sys 0.000s
W20250905 20:00:29.376542 31315 consensus_meta.cc:287] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.808s user 0.012s sys 0.001s
I20250905 20:00:29.554240 28078 raft_consensus.cc:2391] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 8: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:29.277982 31212 raft_consensus.cc:3058] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Advancing to term 9
I20250905 20:00:29.270510 31328 raft_consensus.cc:491] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:29.554842 28715 leader_election.cc:304] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:29.238823 31275 consensus_queue.cc:1046] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.311s
I20250905 20:00:29.378877 31234 raft_consensus.cc:2802] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Leader election won for term 5
I20250905 20:00:29.274680 31190 raft_consensus.cc:2802] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250905 20:00:29.389143 31266 raft_consensus.cc:491] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:29.247120 31388 consensus_queue.cc:260] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:29.557801 31321 raft_consensus.cc:3058] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:29.409274 31203 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:29.429412 31355 raft_consensus.cc:491] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:29.351101 31171 raft_consensus.cc:695] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 11 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:29.537688 28752 rpcz_store.cc:276] 0905 20:00:27.846684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:27.846923 (+ 239us) service_pool.cc:225] Handling call
0905 20:00:27.847398 (+ 475us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:27.847713 (+ 315us) raft_consensus.cc:1532] Early marking committed up to index 7
0905 20:00:28.485661 (+637948us) log.cc:844] Serialized 14 byte log entry
0905 20:00:28.486229 (+ 568us) log.cc:844] Serialized 14 byte log entry
0905 20:00:29.374068 (+887839us) spinlock_profiling.cc:243] Waited 20.7 ms on lock 0x7b18004d58a0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49ac19f4a 00007fe4a4107409 00007fe49aadb3e0 00007fe49aca14e4 00007fe49aca1455 00007fe49dc11020 00007fe49dc236c6 00007fe49dc34237 00007fe49dc341b8
0905 20:00:29.374261 (+ 193us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:29.374322 (+ 61us) raft_consensus.cc:1643] Marking committed up to 7
0905 20:00:29.374382 (+ 60us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:29.374527 (+ 145us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:29.374681 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":873368,"prepare.queue_time_us":654860,"raft.queue_time_us":897620,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":201,"spinlock_wait_cycles":91177088,"thread_start_us":632534,"threads_started":1}
I20250905 20:00:29.541285 31151 raft_consensus.cc:491] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:29.430209 31269 raft_consensus.cc:491] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:29.377966 31313 leader_election.cc:290] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:29.568697 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:29.548439 31263 raft_consensus.cc:491] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:29.549289 31398 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:29.575929 28744 raft_consensus.cc:1238] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 7. Current term is 8. Ops: []
I20250905 20:00:29.548341 31250 raft_consensus.cc:491] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
W20250905 20:00:29.579478 31284 consensus_meta.cc:287] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.630s user 0.013s sys 0.000s
I20250905 20:00:29.582289 28006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 2218) took 1410 ms. Trace:
I20250905 20:00:29.353379 31282 consensus_queue.cc:1046] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.217s
I20250905 20:00:29.586321 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:29.574631 31190 raft_consensus.cc:491] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:29.569880 31328 raft_consensus.cc:3058] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:29.629616 28003 raft_consensus.cc:2391] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 7: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
I20250905 20:00:29.546321 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:29.638141 28079 raft_consensus.cc:3058] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:29.638532 28003 raft_consensus.cc:1238] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:29.639945 28002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 2220) took 1464 ms. Trace:
I20250905 20:00:29.640394 28002 rpcz_store.cc:276] 0905 20:00:28.175693 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:29.205594 (+1029901us) service_pool.cc:225] Handling call
0905 20:00:29.291714 (+ 86120us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:29.390420 (+ 98706us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:29.390477 (+ 57us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:29.420542 (+ 30065us) log.cc:844] Serialized 26 byte log entry
0905 20:00:29.592582 (+172040us) raft_consensus.cc:1643] Marking committed up to 3
0905 20:00:29.592647 (+ 65us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:29.592757 (+ 110us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:29.639706 (+ 46949us) raft_consensus.cc:1683] finished
0905 20:00:29.639731 (+ 25us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:29.639885 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":97976,"raft.queue_time_us":98869,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":20230,"thread_start_us":75181,"threads_started":1,"wal-append.queue_time_us":185965}
I20250905 20:00:29.644129 31284 raft_consensus.cc:513] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:29.571487 31261 raft_consensus.cc:2802] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader election won for term 8
I20250905 20:00:29.634007 28078 raft_consensus.cc:2391] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 6: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:29.555375 31341 raft_consensus.cc:513] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:29.652253 28006 rpcz_store.cc:276] 0905 20:00:28.171829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:29.205774 (+1033945us) service_pool.cc:225] Handling call
0905 20:00:29.240024 (+ 34250us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:29.240463 (+ 439us) raft_consensus.cc:1532] Early marking committed up to index 3
0905 20:00:29.296141 (+ 55678us) log.cc:844] Serialized 14 byte log entry
0905 20:00:29.581757 (+285616us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:29.581847 (+ 90us) raft_consensus.cc:1643] Marking committed up to 3
0905 20:00:29.581921 (+ 74us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:29.582023 (+ 102us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:29.582206 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7678,"prepare.queue_time_us":85967,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":60,"raft.queue_time_us":7842,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":28867,"thread_start_us":276930,"threads_started":1,"wal-append.queue_time_us":348694}
I20250905 20:00:29.569985 31391 raft_consensus.cc:3058] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:29.671908 28744 raft_consensus.cc:1273] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250905 20:00:29.620862 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
W20250905 20:00:29.687561 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 138583us
I20250905 20:00:29.579258 31266 raft_consensus.cc:513] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:29.642485 31190 raft_consensus.cc:3058] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:29.682348 31426 raft_consensus.cc:491] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:29.569458 31315 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:29.686110 28152 raft_consensus.cc:1238] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 LEADER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:29.632511 31388 raft_consensus.cc:3058] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:29.566651 31397 raft_consensus.cc:513] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:29.720410 31397 leader_election.cc:290] T 7eae4eae1373464d85793607943a7c4d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 20:00:29.693554 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 164941us
W20250905 20:00:29.727355 31416 log.cc:927] Time spent T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.150s user 0.001s sys 0.000s
I20250905 20:00:29.730922 31293 raft_consensus.cc:491] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:29.731151 31432 raft_consensus.cc:491] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:29.731653 31293 raft_consensus.cc:513] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:29.732021 31432 raft_consensus.cc:513] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:29.741883 28122 connection.cc:232] RPC call timeout handler was delayed by -1.0209611570001016s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1775s for a response
I20250905 20:00:29.633087 31398 raft_consensus.cc:3058] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Advancing to term 9
I20250905 20:00:29.585825 31407 consensus_queue.cc:1046] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.101s
I20250905 20:00:29.673938 28084 raft_consensus.cc:1238] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:29.710047 28006 raft_consensus.cc:1238] T 0e4ec392d0d54040b0b2162eb0601c0d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 LEADER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 8. Current term is 9. Ops: []
I20250905 20:00:29.712443 28747 raft_consensus.cc:1238] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [term 5 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:29.618291 31355 raft_consensus.cc:513] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:29.713348 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 158564us
I20250905 20:00:29.959532 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:29.960459 28084 raft_consensus.cc:3058] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Advancing to term 7
W20250905 20:00:29.716455 31406 log.cc:927] Time spent T 1d1a8f2db2604a879529dda40b457a4a P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.189s user 0.001s sys 0.000s
W20250905 20:00:29.717283 31410 log.cc:927] Time spent T a002b516fab044b998f136c341a88b74 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.141s user 0.000s sys 0.000s
I20250905 20:00:29.717411 31315 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:29.718057 28155 raft_consensus.cc:1238] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 7. Current term is 8. Ops: []
W20250905 20:00:29.718161 31409 log.cc:927] Time spent T fd2b679c53994263846f51ceec97a150 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.266s user 0.000s sys 0.001s
W20250905 20:00:29.748736 28010 consensus_meta.cc:287] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.628s user 0.008s sys 0.002s
I20250905 20:00:29.763226 28158 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:36164
I20250905 20:00:30.040941 28010 raft_consensus.cc:2466] T 60054c57e2554f32a5ad379a79c2a738 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 8.
I20250905 20:00:30.041429 28158 raft_consensus.cc:604] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Received request to transfer leadership to TS cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:30.041823 28010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1647) took 1258 ms. Trace:
I20250905 20:00:30.041999 28158 raft_consensus.cc:612] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250905 20:00:29.622880 31151 raft_consensus.cc:513] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:29.621658 31269 raft_consensus.cc:513] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:29.693135 31234 raft_consensus.cc:695] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [term 5 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:29.824872 28746 raft_consensus.cc:2391] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 6: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:29.825119 31275 consensus_queue.cc:1046] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250905 20:00:29.826328 31432 leader_election.cc:290] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:29.826613 31341 leader_election.cc:290] T fba42101cd094077aa85e3f9186366e7 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:29.827359 31284 leader_election.cc:290] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:29.827942 31417 consensus_queue.cc:1046] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 20:00:29.692260 31402 raft_consensus.cc:491] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:29.629711 31203 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:29.835754 28006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 2226) took 1098 ms. Trace:
I20250905 20:00:30.160776 28006 rpcz_store.cc:276] 0905 20:00:28.737375 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:29.675131 (+937756us) service_pool.cc:225] Handling call
0905 20:00:29.709790 (+ 34659us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:29.835422 (+125632us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:29.835677 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 20:00:29.889402 27973 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:29.686900 31309 raft_consensus.cc:491] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:29.963533 28079 raft_consensus.cc:2408] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250905 20:00:29.641799 31250 raft_consensus.cc:513] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:29.633334 31263 raft_consensus.cc:513] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:29.704231 28743 raft_consensus.cc:1238] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 8. Current term is 9. Ops: []
I20250905 20:00:29.794100 31171 consensus_queue.cc:237] T 7ca47360fb0f4be6a95a798d0a6d25b8 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:29.995034 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:29.793982 28004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 2223) took 1100 ms. Trace:
I20250905 20:00:30.042315 28010 rpcz_store.cc:276] 0905 20:00:28.783901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:29.100034 (+316133us) service_pool.cc:225] Handling call
0905 20:00:30.041729 (+941695us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:30.270125 28010 raft_consensus.cc:1238] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20250905 20:00:30.270995 28010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 406) took 1386 ms. Trace:
I20250905 20:00:30.271355 28010 rpcz_store.cc:276] 0905 20:00:28.884838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:30.269511 (+1384673us) service_pool.cc:225] Handling call
0905 20:00:30.269896 (+ 385us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:30.270674 (+ 778us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:30.270928 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:30.089480 28084 raft_consensus.cc:2408] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
W20250905 20:00:30.104552 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 411658us
I20250905 20:00:29.796005 31311 raft_consensus.cc:491] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:29.833868 31293 leader_election.cc:290] T 10df9d019f2e4e208f4e23d50f551ad6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:29.795632 31282 raft_consensus.cc:491] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:29.972234 31314 raft_consensus.cc:2747] T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250905 20:00:30.211692 28717 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.005s sys 0.005s
W20250905 20:00:30.217494 31196 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.046s sys 0.067s
I20250905 20:00:29.694365 28083 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 11.
I20250905 20:00:30.238260 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:30.272536 28004 rpcz_store.cc:276] 0905 20:00:28.694036 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:29.552555 (+858519us) service_pool.cc:225] Handling call
0905 20:00:29.552880 (+ 325us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:29.560747 (+ 7867us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:29.560799 (+ 52us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:29.561514 (+ 715us) log.cc:844] Serialized 26 byte log entry
0905 20:00:29.793436 (+231922us) raft_consensus.cc:1643] Marking committed up to 2
0905 20:00:29.793499 (+ 63us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:29.793606 (+ 107us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:29.793677 (+ 71us) raft_consensus.cc:1683] finished
0905 20:00:29.793700 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:29.793893 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":495,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":8860,"thread_start_us":162764,"threads_started":1,"wal-append.queue_time_us":108231}
I20250905 20:00:30.343999 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 20:00:30.364854 31396 log.cc:927] Time spent T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a: Append to log took a long time: real 0.142s user 0.001s sys 0.000s
I20250905 20:00:29.794724 31426 raft_consensus.cc:513] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:30.565728 28083 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1723) took 1291 ms. Trace:
I20250905 20:00:30.569259 28083 rpcz_store.cc:276] 0905 20:00:29.277354 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:29.280397 (+ 3043us) service_pool.cc:225] Handling call
0905 20:00:30.565645 (+1285248us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 20:00:30.369529 28717 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.726s user 0.006s sys 0.006s
W20250905 20:00:30.572816 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 929579us
I20250905 20:00:30.575398 28159 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2338) took 2670 ms. Trace:
W20250905 20:00:30.576938 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 300977us
W20250905 20:00:30.606379 28081 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.632s user 0.013s sys 0.002s
I20250905 20:00:30.565022 28748 raft_consensus.cc:2466] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
W20250905 20:00:30.606870 28081 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.809s user 0.013s sys 0.002s
I20250905 20:00:30.592576 28004 raft_consensus.cc:1238] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20250905 20:00:30.607607 28081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1722) took 1333 ms. Trace:
I20250905 20:00:30.608137 28081 rpcz_store.cc:276] 0905 20:00:29.274208 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:29.276466 (+ 2258us) service_pool.cc:225] Handling call
0905 20:00:29.541970 (+265504us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:29.542466 (+ 496us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:29.542500 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:29.551528 (+ 9028us) log.cc:844] Serialized 26 byte log entry
0905 20:00:29.679325 (+127797us) raft_consensus.cc:1643] Marking committed up to 6
0905 20:00:29.714106 (+ 34781us) log.cc:844] Serialized 14 byte log entry
0905 20:00:30.607258 (+893152us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:30.607355 (+ 97us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:30.607407 (+ 52us) raft_consensus.cc:1683] finished
0905 20:00:30.607419 (+ 12us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:30.607552 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":39273,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":101,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":7164,"thread_start_us":919627,"threads_started":3,"wal-append.queue_time_us":110074}
W20250905 20:00:30.544011 28715 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.000s sys 0.012s
W20250905 20:00:30.611630 28715 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.622s user 0.000s sys 0.012s
W20250905 20:00:30.612035 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 653121us
W20250905 20:00:30.258225 28125 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.614s user 0.016s sys 0.025s
W20250905 20:00:30.613822 28125 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.970s user 0.016s sys 0.027s
W20250905 20:00:30.614223 28125 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 970407us
I20250905 20:00:30.602134 28006 raft_consensus.cc:1238] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:30.603386 28159 rpcz_store.cc:276] 0905 20:00:27.905019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:28.824849 (+919830us) service_pool.cc:225] Handling call
0905 20:00:28.831626 (+ 6777us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:28.948213 (+116587us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:28.948264 (+ 51us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:28.948931 (+ 667us) log.cc:844] Serialized 26 byte log entry
0905 20:00:29.147801 (+198870us) raft_consensus.cc:1643] Marking committed up to 3
0905 20:00:29.147846 (+ 45us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:29.372098 (+224252us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b5400187c54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:29.372157 (+ 59us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:30.548817 (+1176660us) raft_consensus.cc:1683] finished
0905 20:00:30.548846 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:30.575304 (+ 26458us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":587,"raft.run_cpu_time_us":64,"raft.run_wall_time_us":63,"spinlock_wait_cycles":266877824,"thread_start_us":177460,"threads_started":1,"wal-append.queue_time_us":516662}
I20250905 20:00:30.620685 28004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 408) took 1730 ms. Trace:
I20250905 20:00:30.621465 28004 rpcz_store.cc:276] 0905 20:00:28.890077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:30.591675 (+1701598us) service_pool.cc:225] Handling call
0905 20:00:30.592241 (+ 566us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:30.620263 (+ 28022us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:30.620593 (+ 330us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:30.458473 28010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 407) took 1573 ms. Trace:
W20250905 20:00:30.581152 28122 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.680s user 0.010s sys 0.005s
W20250905 20:00:30.625912 28122 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.725s user 0.011s sys 0.005s
W20250905 20:00:30.626581 28122 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 725848us
I20250905 20:00:30.625823 28010 rpcz_store.cc:276] 0905 20:00:28.885146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:30.272208 (+1387062us) service_pool.cc:225] Handling call
0905 20:00:30.457716 (+185508us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:30.458029 (+ 313us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:30.458066 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:30.458124 (+ 58us) raft_consensus.cc:1643] Marking committed up to 6
0905 20:00:30.458157 (+ 33us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:30.458209 (+ 52us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:30.458365 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:30.636838 28006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1644) took 1899 ms. Trace:
I20250905 20:00:30.637636 28006 rpcz_store.cc:276] 0905 20:00:28.737453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:30.198249 (+1460796us) service_pool.cc:225] Handling call
0905 20:00:30.601894 (+403645us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:30.636438 (+ 34544us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:30.636773 (+ 335us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:30.612993 28715 leader_election.cc:304] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 20:00:30.564260 31196 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.872s user 0.053s sys 0.080s
W20250905 20:00:30.258507 31261 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.566s user 0.015s sys 0.025s
W20250905 20:00:30.258450 28716 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.012s sys 0.023s
I20250905 20:00:30.304796 31432 raft_consensus.cc:491] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:30.745297 31432 raft_consensus.cc:513] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:30.409194 31311 raft_consensus.cc:513] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:30.229521 31213 consensus_meta.cc:287] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 1.457s user 0.014s sys 0.000s
W20250905 20:00:30.689533 28002 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.994s user 0.038s sys 0.067s
W20250905 20:00:30.756938 28002 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.062s user 0.038s sys 0.068s
I20250905 20:00:30.758028 28002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 2225) took 2028 ms. Trace:
I20250905 20:00:30.758687 28002 rpcz_store.cc:276] 0905 20:00:28.729530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:29.642275 (+912745us) service_pool.cc:225] Handling call
0905 20:00:29.694461 (+ 52186us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:29.694682 (+ 221us) raft_consensus.cc:1532] Early marking committed up to index 4
0905 20:00:29.694973 (+ 291us) log.cc:844] Serialized 14 byte log entry
0905 20:00:30.757574 (+1062601us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:30.757645 (+ 71us) raft_consensus.cc:1643] Marking committed up to 4
0905 20:00:30.757711 (+ 66us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:30.757793 (+ 82us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:30.757946 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":942,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":55,"thread_start_us":1062170,"threads_started":1}
W20250905 20:00:30.692387 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 72410.5us
W20250905 20:00:30.690835 28077 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.702s user 0.000s sys 0.016s
W20250905 20:00:30.760849 28077 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.046s user 0.000s sys 0.017s
W20250905 20:00:30.709125 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:30.227648 31309 raft_consensus.cc:513] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:30.761583 28077 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1719) took 1490 ms. Trace:
I20250905 20:00:30.745975 31250 leader_election.cc:290] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:30.708124 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:30.711045 30946 raft_consensus.cc:491] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
W20250905 20:00:30.715576 31261 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.013s user 0.018s sys 0.025s
I20250905 20:00:30.146375 31422 raft_consensus.cc:2802] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250905 20:00:30.714862 28716 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.980s user 0.012s sys 0.026s
I20250905 20:00:30.693866 28004 raft_consensus.cc:1238] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20250905 20:00:30.761933 28077 rpcz_store.cc:276] 0905 20:00:29.271178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:29.271612 (+ 434us) service_pool.cc:225] Handling call
0905 20:00:29.357932 (+ 86320us) raft_consensus.cc:1469] Updating replica for 2 ops
0905 20:00:29.358473 (+ 541us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:29.358516 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0905 20:00:29.543393 (+184877us) log.cc:844] Serialized 52 byte log entry
0905 20:00:29.683125 (+139732us) raft_consensus.cc:1643] Marking committed up to 4
0905 20:00:29.706928 (+ 23803us) log.cc:844] Serialized 14 byte log entry
0905 20:00:30.761248 (+1054320us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:30.761328 (+ 80us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:30.761376 (+ 48us) raft_consensus.cc:1683] finished
0905 20:00:30.761387 (+ 11us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:30.761527 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23367,"prepare.queue_time_us":35691,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":871359,"raft.run_cpu_time_us":376,"raft.run_wall_time_us":52790,"thread_start_us":1174510,"threads_started":3,"wal-append.queue_time_us":120705,"wal-append.run_cpu_time_us":3033,"wal-append.run_wall_time_us":1037408}
I20250905 20:00:30.708542 28010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 410) took 1817 ms. Trace:
W20250905 20:00:30.774675 27973 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.580s user 0.034s sys 0.070s
I20250905 20:00:30.622200 31315 raft_consensus.cc:491] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:30.523990 31282 raft_consensus.cc:513] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:30.787467 28049 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.941s user 0.042s sys 0.055s
W20250905 20:00:30.683775 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:30.803746 27973 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.609s user 0.034s sys 0.070s
I20250905 20:00:30.803627 28010 rpcz_store.cc:276] 0905 20:00:28.890953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:30.690537 (+1799584us) service_pool.cc:225] Handling call
0905 20:00:30.707894 (+ 17357us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:30.708123 (+ 229us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:30.708147 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:30.708198 (+ 51us) raft_consensus.cc:1643] Marking committed up to 9
0905 20:00:30.708246 (+ 48us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:30.708319 (+ 73us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:30.708479 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:30.803649 28004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 409) took 1913 ms. Trace:
I20250905 20:00:30.813107 28004 rpcz_store.cc:276] 0905 20:00:28.890605 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:30.692999 (+1802394us) service_pool.cc:225] Handling call
0905 20:00:30.693540 (+ 541us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:30.803283 (+109743us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:30.803592 (+ 309us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:30.812201 28002 raft_consensus.cc:3053] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 LEADER]: Stepping down as leader of term 8
I20250905 20:00:30.814240 28002 raft_consensus.cc:738] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
W20250905 20:00:30.815001 28049 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.968s user 0.043s sys 0.055s
I20250905 20:00:30.194074 31402 raft_consensus.cc:513] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:30.841315 28004 raft_consensus.cc:1238] T 0860997b3256400084f5884190bbe6e2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:30.779650 31213 raft_consensus.cc:2747] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250905 20:00:30.843978 28004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 2227) took 1296 ms. Trace:
I20250905 20:00:30.844447 28004 rpcz_store.cc:276] 0905 20:00:29.547162 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:30.840735 (+1293573us) service_pool.cc:225] Handling call
0905 20:00:30.841110 (+ 375us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:30.843666 (+ 2556us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:30.843913 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:30.817951 28002 consensus_queue.cc:260] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:30.845921 28002 raft_consensus.cc:3058] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Advancing to term 9
I20250905 20:00:30.824326 28006 raft_consensus.cc:1273] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 10 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20250905 20:00:30.848968 28006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45844 (request call id 411) took 1957 ms. Trace:
I20250905 20:00:30.849642 28006 rpcz_store.cc:276] 0905 20:00:28.891479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:30.690708 (+1799229us) service_pool.cc:225] Handling call
0905 20:00:30.816100 (+125392us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:30.848501 (+ 32401us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:30.848868 (+ 367us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:30.851722 28743 raft_consensus.cc:1273] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20250905 20:00:30.852109 31426 leader_election.cc:290] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:30.840708 31299 raft_consensus.cc:491] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:30.853487 31426 raft_consensus.cc:491] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:30.853252 31299 raft_consensus.cc:513] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:30.854705 31210 consensus_meta.cc:287] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a: Time spent flushing consensus metadata: real 1.751s user 0.011s sys 0.002s
I20250905 20:00:30.855126 31210 raft_consensus.cc:513] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:30.855612 31299 leader_election.cc:290] T d06337b10d154efbbaaf3c152d01f6dd P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:30.856253 31299 raft_consensus.cc:491] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:30.736922 31234 consensus_queue.cc:237] T 2f356a2522e646889c023d4b3d26f9cc P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:30.857369 31210 leader_election.cc:290] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:30.855194 28010 raft_consensus.cc:1273] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20250905 20:00:30.860133 28010 pending_rounds.cc:85] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Aborting all ops after (but not including) 7
I20250905 20:00:30.860428 28010 pending_rounds.cc:107] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20250905 20:00:30.860697 28010 raft_consensus.cc:2887] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250905 20:00:30.861568 28010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1656) took 1886 ms. Trace:
I20250905 20:00:30.861972 28010 rpcz_store.cc:276] 0905 20:00:28.975096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:30.840291 (+1865195us) service_pool.cc:225] Handling call
0905 20:00:30.854841 (+ 14550us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:30.861238 (+ 6397us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:30.861497 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:30.856805 31299 raft_consensus.cc:513] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:30.796876 31261 raft_consensus.cc:695] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 LEADER]: Becoming Leader. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:30.802462 31422 raft_consensus.cc:491] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250905 20:00:30.854348 31212 consensus_meta.cc:287] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 1.283s user 0.007s sys 0.000s
I20250905 20:00:30.819290 31293 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:30.796537 31250 raft_consensus.cc:491] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:30.814603 31266 leader_election.cc:290] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:30.854029 31426 raft_consensus.cc:513] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:30.867260 31309 leader_election.cc:290] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:30.867125 31439 raft_consensus.cc:491] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:30.880880 31298 raft_consensus.cc:2802] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader election won for term 8
I20250905 20:00:30.873724 31440 raft_consensus.cc:3053] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Stepping down as leader of term 4
I20250905 20:00:30.818965 31315 raft_consensus.cc:513] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:30.867918 31427 raft_consensus.cc:491] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
W20250905 20:00:30.955145 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:31.007575 31425 raft_consensus.cc:491] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:30.880681 31355 leader_election.cc:290] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:30.889254 31151 leader_election.cc:290] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:30.806910 30946 raft_consensus.cc:513] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
W20250905 20:00:31.023442 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 262762us
I20250905 20:00:30.950362 31266 raft_consensus.cc:491] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:30.907428 31449 raft_consensus.cc:491] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:30.919770 31203 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:31.051739 28717 leader_election.cc:304] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:31.021832 31440 raft_consensus.cc:738] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:30.942276 31429 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:31.031563 31298 raft_consensus.cc:695] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 LEADER]: Becoming Leader. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:31.097805 31217 raft_consensus.cc:491] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:30.949754 31397 raft_consensus.cc:491] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:31.098889 31397 raft_consensus.cc:513] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:31.101598 31397 leader_election.cc:290] T b13af7609cd04cf5b21745527b70074d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:31.008991 31269 leader_election.cc:290] T 24cf07baf43247b6bf63063106b79836 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:31.021587 31263 leader_election.cc:290] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:30.908857 31422 raft_consensus.cc:3058] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:31.069811 31449 raft_consensus.cc:513] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:30.941586 31454 raft_consensus.cc:491] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:31.046838 31427 raft_consensus.cc:513] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.077731 31429 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:31.022241 31439 raft_consensus.cc:513] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:31.042874 31311 leader_election.cc:290] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:31.079241 28152 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2319) took 4390 ms. Trace:
I20250905 20:00:31.047454 31457 raft_consensus.cc:491] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:31.081039 31315 leader_election.cc:290] T 10dd790c7b5c4cc5978c3545b5a06868 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:31.081117 28748 raft_consensus.cc:1238] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 7. Current term is 8. Ops: []
I20250905 20:00:31.083364 31299 leader_election.cc:290] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:31.102365 28747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40904 (request call id 792) took 1461 ms. Trace:
I20250905 20:00:31.123248 28747 rpcz_store.cc:276] 0905 20:00:29.640961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:29.641102 (+ 141us) service_pool.cc:225] Handling call
0905 20:00:29.651780 (+ 10678us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:29.835582 (+183802us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:31.101999 (+1266417us) spinlock_profiling.cc:243] Waited 60.3 ms on lock 0x7b54004b0d54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:31.102269 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":138803072}
I20250905 20:00:31.105361 28157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33752 (request call id 1760) took 3525 ms. Trace:
I20250905 20:00:31.125267 28157 rpcz_store.cc:276] 0905 20:00:27.580015 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:28.552012 (+971997us) service_pool.cc:225] Handling call
0905 20:00:28.552416 (+ 404us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:29.065687 (+513271us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:31.105050 (+2039363us) spinlock_profiling.cc:243] Waited 442 ms on lock 0x7b54000eced4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:31.105258 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1016109952}
W20250905 20:00:31.085590 31190 consensus_meta.cc:287] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.378s user 0.006s sys 0.006s
I20250905 20:00:31.087273 30946 leader_election.cc:290] T bbd67b7a3c3244838ec2d727f746cc0b P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 20:00:31.089980 31415 log.cc:927] Time spent T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 1.467s user 0.005s sys 0.019s
I20250905 20:00:31.091218 31402 leader_election.cc:290] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:31.092088 31282 leader_election.cc:290] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:31.092695 31328 consensus_meta.cc:287] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 1.422s user 0.009s sys 0.003s
I20250905 20:00:31.092643 31432 leader_election.cc:290] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:31.093266 31196 raft_consensus.cc:491] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:31.055418 31355 raft_consensus.cc:491] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:31.116153 31439 leader_election.cc:290] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:31.095978 31321 consensus_meta.cc:287] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 1.476s user 0.002s sys 0.010s
I20250905 20:00:31.053387 31425 raft_consensus.cc:513] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:30.939522 31453 raft_consensus.cc:491] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:31.074891 31203 raft_consensus.cc:2747] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20250905 20:00:31.122411 31388 consensus_meta.cc:287] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.387s user 0.005s sys 0.005s
I20250905 20:00:31.195379 31196 raft_consensus.cc:513] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:31.254053 28150 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39586 (request call id 429) took 3182 ms. Trace:
I20250905 20:00:31.111415 31217 raft_consensus.cc:513] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:31.281735 28150 rpcz_store.cc:276] 0905 20:00:28.071865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:28.884270 (+812405us) service_pool.cc:225] Handling call
0905 20:00:28.945946 (+ 61676us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:28.946487 (+ 541us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:28.946534 (+ 47us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:28.947160 (+ 626us) log.cc:844] Serialized 26 byte log entry
0905 20:00:29.123408 (+176248us) raft_consensus.cc:1643] Marking committed up to 7
0905 20:00:29.123468 (+ 60us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:29.123564 (+ 96us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:31.126402 (+2002838us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b24001812c8. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49aaf9784 00007fe49aaf788b 00007fe49ab86669 00007fe49ab862f5 00007fe49ab85850 00007fe49d885498 00007fe49d885bf8 00007fe49d85d438 00007fe49d85881a 00007fe49d857c66
0905 20:00:31.252902 (+126500us) raft_consensus.cc:1683] finished
0905 20:00:31.252995 (+ 93us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:31.253667 (+ 672us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2043,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":143,"spinlock_wait_cycles":34052352,"thread_start_us":175773,"threads_started":1,"wal-append.queue_time_us":375243}
I20250905 20:00:31.284195 31276 raft_consensus.cc:491] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:30.915097 31250 raft_consensus.cc:513] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:31.286980 31312 raft_consensus.cc:491] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:31.288595 31312 raft_consensus.cc:513] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.289314 31250 leader_election.cc:290] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:31.064579 31266 raft_consensus.cc:513] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:31.191429 31328 raft_consensus.cc:513] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.116658 31440 consensus_queue.cc:260] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.292562 31440 raft_consensus.cc:3058] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:31.294930 31269 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:31.296213 31269 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:31.215576 31408 log.cc:927] Time spent T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 1.805s user 0.010s sys 0.017s
I20250905 20:00:31.064216 31261 consensus_queue.cc:237] T 60054c57e2554f32a5ad379a79c2a738 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.299130 31282 raft_consensus.cc:491] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:31.300704 31282 raft_consensus.cc:513] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:30.914628 31293 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.199539 31284 raft_consensus.cc:491] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:31.303871 31284 raft_consensus.cc:513] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.097807 31298 consensus_queue.cc:237] T 05b2fd124328472aa3e3f62e0b327582 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:31.305689 31293 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:31.116158 31429 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:31.213991 31433 raft_consensus.cc:3053] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 LEADER]: Stepping down as leader of term 5
W20250905 20:00:31.114118 31383 consensus_meta.cc:287] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.563s user 0.009s sys 0.001s
I20250905 20:00:31.307523 31433 raft_consensus.cc:738] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:31.308111 31383 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:31.033090 28051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 416579us
W20250905 20:00:31.200626 31391 consensus_meta.cc:287] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.506s user 0.010s sys 0.003s
I20250905 20:00:31.217018 31425 leader_election.cc:290] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:31.311105 28003 raft_consensus.cc:1238] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:31.121219 31339 raft_consensus.cc:3053] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 7 LEADER]: Stepping down as leader of term 7
I20250905 20:00:31.355360 31339 raft_consensus.cc:738] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:31.357362 31339 consensus_queue.cc:260] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:31.118973 31299 raft_consensus.cc:491] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
W20250905 20:00:31.383381 31390 log.cc:927] Time spent T 456d9fe0918b423abeab5c39372bcbe9 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 2.502s user 0.004s sys 0.029s
I20250905 20:00:31.384342 31299 raft_consensus.cc:513] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.148087 31422 raft_consensus.cc:513] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:31.395320 31339 raft_consensus.cc:3058] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:31.395612 31306 raft_consensus.cc:491] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:31.396313 31306 raft_consensus.cc:3058] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:31.397305 31422 leader_election.cc:290] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:31.112694 31427 leader_election.cc:290] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:31.131675 31397 raft_consensus.cc:491] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:31.399386 31397 raft_consensus.cc:513] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:31.400272 31299 leader_election.cc:290] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 20:00:31.136873 31398 consensus_meta.cc:287] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 1.164s user 0.005s sys 0.006s
W20250905 20:00:31.403501 28008 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1617) took 3162 ms (client timeout 1940 ms). Trace:
I20250905 20:00:31.403369 31398 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:31.404032 28008 rpcz_store.cc:269] 0905 20:00:28.241268 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:31.402947 (+3161679us) service_pool.cc:212] Skipping call since client already timed out
0905 20:00:31.403222 (+ 275us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250905 20:00:31.405647 28005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54156 (request call id 2216) took 3237 ms. Trace:
W20250905 20:00:31.132412 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:31.406147 28005 rpcz_store.cc:276] 0905 20:00:28.168368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:28.168530 (+ 162us) service_pool.cc:225] Handling call
0905 20:00:28.169005 (+ 475us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:28.206680 (+ 37675us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:28.206728 (+ 48us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:28.239175 (+ 32447us) log.cc:844] Serialized 26 byte log entry
0905 20:00:28.718127 (+478952us) raft_consensus.cc:1643] Marking committed up to 4
0905 20:00:28.769529 (+ 51402us) log.cc:844] Serialized 14 byte log entry
0905 20:00:28.769792 (+ 263us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:28.769887 (+ 95us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:29.721115 (+951228us) spinlock_profiling.cc:243] Waited 12.8 ms on lock 0x7b18004eb980. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49aca43e2 00007fe49aca67b1 00007fe49aca6769 00007fe49aca66f9 00007fe49aca66c1 00007fe49aca59bd 0000000000379254 0000000000379099 00007fe49ac8776a
0905 20:00:31.382211 (+1661096us) spinlock_profiling.cc:243] Waited 4.69 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49d85891e 00007fe49d857c66 00007fe49d857888 00007fe49d8612c1 00007fe49d861279 00007fe49d861209
0905 20:00:31.386655 (+ 4444us) raft_consensus.cc:1683] finished
0905 20:00:31.386680 (+ 25us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:31.405403 (+ 18723us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37019,"prepare.queue_time_us":54415,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":75,"raft.queue_time_us":44254,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":207,"spinlock_wait_cycles":40289024,"thread_start_us":427471,"threads_started":1,"wal-append.queue_time_us":573253}
I20250905 20:00:31.409018 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:31.410250 28747 raft_consensus.cc:2391] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 9: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:31.117326 28152 rpcz_store.cc:276] 0905 20:00:26.688648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:26.688877 (+ 229us) service_pool.cc:225] Handling call
0905 20:00:26.717188 (+ 28311us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:29.707174 (+2989986us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:31.078949 (+1371775us) spinlock_profiling.cc:243] Waited 2.97 s on lock 0x7b5400187754. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:31.079157 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6827085184}
I20250905 20:00:31.114326 28746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:31.416227 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:31.416783 28746 raft_consensus.cc:2408] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20250905 20:00:31.417217 28751 raft_consensus.cc:2466] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
I20250905 20:00:31.176458 31190 raft_consensus.cc:513] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.263167 28157 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:36164
I20250905 20:00:31.418704 28157 raft_consensus.cc:604] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Received request to transfer leadership to TS cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:31.419102 28157 raft_consensus.cc:612] T 456d9fe0918b423abeab5c39372bcbe9 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250905 20:00:31.109624 31441 log.cc:927] Time spent T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.158s user 0.000s sys 0.000s
W20250905 20:00:31.110688 31447 log.cc:927] Time spent T 00000000000000000000000000000000 P 44c052c5498545aa93c222520045d410: Append to log took a long time: real 0.243s user 0.000s sys 0.002s
I20250905 20:00:31.108109 31449 leader_election.cc:290] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:31.201184 31453 raft_consensus.cc:513] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.432482 28157 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5f31007f29044bd1817fcf19ba904ad3"
dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
mode: GRACEFUL
new_leader_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0"
from {username='slave'} at 127.0.0.1:36164
I20250905 20:00:31.110908 31454 raft_consensus.cc:513] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:31.117578 31457 raft_consensus.cc:513] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:31.104388 31461 raft_consensus.cc:491] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:31.197731 31355 raft_consensus.cc:513] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:31.258942 28123 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.519s user 0.035s sys 0.027s
I20250905 20:00:31.269634 28752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40926 (request call id 560) took 1035 ms. Trace:
I20250905 20:00:31.285568 31276 raft_consensus.cc:513] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.294100 31328 leader_election.cc:290] T 6c423052d6454829bccfe7c8bc1bda5c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:31.348168 27973 leader_election.cc:304] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:31.293538 31266 leader_election.cc:290] T 8e0ac2dfe59d423189dd8d72eee98a42 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:31.350569 31388 raft_consensus.cc:491] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:31.352779 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:31.354568 31217 leader_election.cc:290] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:31.358070 31383 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 11 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:31.359727 31084 consensus_queue.cc:1046] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20250905 20:00:31.361053 31402 raft_consensus.cc:491] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:31.117939 31315 raft_consensus.cc:491] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:31.364764 31269 leader_election.cc:290] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:31.366885 31455 consensus_queue.cc:1046] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250905 20:00:31.367828 31312 leader_election.cc:290] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:31.367937 31426 leader_election.cc:290] T 62ccdb07048e4762b1b31c36b14610f5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:31.368561 28001 raft_consensus.cc:2466] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 20:00:31.354645 31435 raft_consensus.cc:491] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:31.370556 31196 leader_election.cc:290] T 05b2fd124328472aa3e3f62e0b327582 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:31.370972 28049 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:31.371228 28003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1643) took 2659 ms. Trace:
W20250905 20:00:31.292654 28082 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59564 (request call id 1702) took 3037 ms (client timeout 1775 ms). Trace:
I20250905 20:00:31.505957 31084 raft_consensus.cc:3053] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 LEADER]: Stepping down as leader of term 9
I20250905 20:00:31.371258 28085 raft_consensus.cc:2466] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:31.376870 28153 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2353) took 2562 ms. Trace:
I20250905 20:00:31.411167 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6a8d5d253643cea8582bef1c4173d6" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:31.115154 28744 raft_consensus.cc:1273] T c0889204295e4daa99d81e5b2a45ac1c P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20250905 20:00:31.512756 28082 rpcz_store.cc:269] 0905 20:00:28.260628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:28.261506 (+ 878us) service_pool.cc:225] Handling call
0905 20:00:31.291918 (+3030412us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b54003bf6d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:31.292358 (+ 440us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":522582912}
I20250905 20:00:31.513850 31306 raft_consensus.cc:513] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.523938 31276 leader_election.cc:290] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:31.420339 31190 leader_election.cc:290] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:31.524981 31190 raft_consensus.cc:491] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:31.525279 31325 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:31.514110 31084 raft_consensus.cc:738] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:31.525696 31190 raft_consensus.cc:513] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:31.526629 31427 raft_consensus.cc:491] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:31.527129 31084 consensus_queue.cc:260] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.527383 31427 raft_consensus.cc:513] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:31.528079 31454 leader_election.cc:290] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:31.528203 31190 leader_election.cc:290] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:31.528605 31084 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Advancing to term 10
I20250905 20:00:31.528906 31190 raft_consensus.cc:491] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:31.528991 31461 raft_consensus.cc:513] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:31.529882 31427 leader_election.cc:290] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:31.531625 28152 raft_consensus.cc:1238] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 LEADER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 8. Current term is 9. Ops: []
I20250905 20:00:31.531809 31461 leader_election.cc:290] T 8afd77a8d80843aa805f971a2806ab86 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:31.532455 31461 raft_consensus.cc:491] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:31.533299 31461 raft_consensus.cc:513] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.309648 31433 consensus_queue.cc:260] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.513119 28153 rpcz_store.cc:276] 0905 20:00:28.814355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:29.087885 (+273530us) service_pool.cc:225] Handling call
0905 20:00:29.088235 (+ 350us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:31.281840 (+2193605us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:31.281932 (+ 92us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:31.282001 (+ 69us) raft_consensus.cc:1643] Marking committed up to 4
0905 20:00:31.282121 (+ 120us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:31.376366 (+ 94245us) spinlock_profiling.cc:243] Waited 2.19 s on lock 0x7b54000ec9d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:31.376432 (+ 66us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:31.376677 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":16413,"raft.run_cpu_time_us":364,"raft.run_wall_time_us":365,"spinlock_wait_cycles":5042594048}
I20250905 20:00:31.536619 28159 raft_consensus.cc:1273] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250905 20:00:31.529678 31190 raft_consensus.cc:513] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.424014 31397 leader_election.cc:290] T 9f7f6e4e3a754dcda90928621168fbe7 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:31.386814 31282 leader_election.cc:290] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:31.434976 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:31.425930 31422 raft_consensus.cc:491] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:31.537961 28751 raft_consensus.cc:2466] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 7.
I20250905 20:00:31.538079 31422 raft_consensus.cc:513] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:31.538777 31455 consensus_queue.cc:1230] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer cd764bfeebac4cc9ad851cd4a6a7d3b0 log is divergent from this leader: its last log entry 9.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250905 20:00:31.438735 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:31.541599 28747 raft_consensus.cc:2466] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 7.
I20250905 20:00:31.428220 31398 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:31.511351 31388 raft_consensus.cc:513] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.543644 31422 leader_election.cc:290] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 20:00:31.481480 27974 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.878s user 0.005s sys 0.009s
I20250905 20:00:31.544366 28748 raft_consensus.cc:2391] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 12: Already voted for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in this term.
W20250905 20:00:31.544660 27974 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.941s user 0.006s sys 0.010s
W20250905 20:00:31.545184 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 941872us
I20250905 20:00:31.515259 31266 raft_consensus.cc:2747] T a002b516fab044b998f136c341a88b74 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250905 20:00:31.489763 28752 rpcz_store.cc:276] 0905 20:00:30.234409 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:30.305127 (+ 70718us) service_pool.cc:225] Handling call
0905 20:00:30.532021 (+226894us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:30.532534 (+ 513us) raft_consensus.cc:1532] Early marking committed up to index 6
0905 20:00:30.532576 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:30.533180 (+ 604us) log.cc:844] Serialized 26 byte log entry
0905 20:00:30.951117 (+417937us) raft_consensus.cc:1643] Marking committed up to 6
0905 20:00:30.951166 (+ 49us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:30.951242 (+ 76us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:31.268695 (+317453us) raft_consensus.cc:1683] finished
0905 20:00:31.268766 (+ 71us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:31.269243 (+ 477us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":52467,"raft.queue_time_us":285839,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":25005,"thread_start_us":253336,"threads_started":1,"wal-append.queue_time_us":607195}
I20250905 20:00:31.549031 31439 raft_consensus.cc:491] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:31.550251 31439 raft_consensus.cc:513] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.551756 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
W20250905 20:00:31.552852 28749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40904 (request call id 789) took 1976 ms (client timeout 2213 ms). Trace:
I20250905 20:00:31.486058 31224 raft_consensus.cc:491] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:31.553231 31439 leader_election.cc:290] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:31.553692 31224 raft_consensus.cc:513] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:31.549093 28049 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.26.58.132:44299 timed out after 1.665s (SENT) [suppressed 34 similar messages]
W20250905 20:00:31.481444 31332 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.026s sys 0.052s
W20250905 20:00:31.556296 31332 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.616s user 0.026s sys 0.053s
I20250905 20:00:31.556703 31224 leader_election.cc:290] T aace14abd237420c8211ce611ea09f0e P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:31.508688 31453 leader_election.cc:290] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:31.505762 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:31.512653 28003 rpcz_store.cc:276] 0905 20:00:28.712263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:29.639556 (+927293us) service_pool.cc:225] Handling call
0905 20:00:29.639887 (+ 331us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:31.311883 (+1671996us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:31.370774 (+ 58891us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b54003836d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:31.371060 (+ 286us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3841714560}
I20250905 20:00:31.520079 31463 raft_consensus.cc:491] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:31.519910 31315 raft_consensus.cc:513] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.526305 31325 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:31.562690 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:31.521200 31402 raft_consensus.cc:513] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.573122 31402 leader_election.cc:290] T 10dd790c7b5c4cc5978c3545b5a06868 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
W20250905 20:00:31.420380 28080 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1724) took 2139 ms (client timeout 1957 ms). Trace:
W20250905 20:00:31.573973 28080 rpcz_store.cc:269] 0905 20:00:29.280777 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:31.419920 (+2139143us) service_pool.cc:212] Skipping call since client already timed out
0905 20:00:31.420149 (+ 229us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250905 20:00:31.440186 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:31.549338 31220 raft_consensus.cc:491] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:31.536242 31435 raft_consensus.cc:513] T 18c49d7ed5d64445afa2c6ee7a2e27d1 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.579439 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:31.513954 31217 consensus_queue.cc:1046] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.067s
W20250905 20:00:31.495568 28123 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.756s user 0.036s sys 0.028s
W20250905 20:00:31.526300 28002 consensus_meta.cc:287] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.680s user 0.014s sys 0.000s
W20250905 20:00:31.507236 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 233916us
I20250905 20:00:31.521545 31457 leader_election.cc:290] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:31.572945 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:31.574725 28005 raft_consensus.cc:1273] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250905 20:00:31.525475 31306 leader_election.cc:290] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:31.599270 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 20:00:31.559267 28749 rpcz_store.cc:269] 0905 20:00:29.576252 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:29.576440 (+ 188us) service_pool.cc:225] Handling call
0905 20:00:31.551394 (+1974954us) spinlock_profiling.cc:243] Waited 5.12 ms on lock 0x7b5800304598. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b7db60 00007fe4a2b7d444 00007fe49c2050c4 00007fe49c205052
0905 20:00:31.552765 (+ 1371us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11775360}
I20250905 20:00:31.575497 31220 raft_consensus.cc:513] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:31.560956 31452 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.540s user 0.023s sys 0.057s
I20250905 20:00:31.557544 31355 leader_election.cc:290] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:31.587527 31325 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:31.591043 31461 leader_election.cc:290] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:31.591269 31403 raft_consensus.cc:3053] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Stepping down as leader of term 5
I20250905 20:00:31.592725 31261 raft_consensus.cc:491] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:31.594681 28006 raft_consensus.cc:2466] T 361ffd5c9a994e0397e6748827466d3d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 7.
I20250905 20:00:31.594916 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:31.595999 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:31.594754 28002 raft_consensus.cc:1273] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 9. (term mismatch)
I20250905 20:00:31.596231 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:31.668123 28159 raft_consensus.cc:3058] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:31.596565 28004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bd6bb972f14472a89ccf99de880efc2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:31.563073 31427 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:31.596939 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:31.598228 31455 raft_consensus.cc:3053] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 LEADER]: Stepping down as leader of term 6
I20250905 20:00:31.598170 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2b679c53994263846f51ceec97a150" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:31.638255 31357 raft_consensus.cc:491] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:31.641119 31220 leader_election.cc:290] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:31.645896 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:31.647609 31463 raft_consensus.cc:513] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:31.648495 31458 raft_consensus.cc:491] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:31.648793 31355 raft_consensus.cc:491] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:31.650094 31457 raft_consensus.cc:491] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:31.651691 31171 consensus_queue.cc:1035] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250905 20:00:31.652561 31426 raft_consensus.cc:491] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:31.654294 31252 raft_consensus.cc:3053] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 8 LEADER]: Stepping down as leader of term 8
I20250905 20:00:31.705780 31252 raft_consensus.cc:738] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:31.707125 31252 consensus_queue.cc:260] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.708775 31252 raft_consensus.cc:3058] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Advancing to term 9
I20250905 20:00:31.709021 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:31.709882 28154 raft_consensus.cc:2466] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 7.
W20250905 20:00:31.686734 28001 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54164 (request call id 1623) took 3401 ms (client timeout 2499 ms). Trace:
W20250905 20:00:31.711347 28001 rpcz_store.cc:269] 0905 20:00:28.280393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:29.096775 (+816382us) service_pool.cc:225] Handling call
0905 20:00:31.665502 (+2568727us) spinlock_profiling.cc:243] Waited 2.21 s on lock 0x7b5400384fd4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:31.682060 (+ 16558us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5093281536}
I20250905 20:00:31.712324 28151 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2339) took 3806 ms. Trace:
I20250905 20:00:31.712972 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b18b7149ff3439a8dde09149cf336b3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:31.714124 28743 raft_consensus.cc:2466] T 1b18b7149ff3439a8dde09149cf336b3 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 8.
I20250905 20:00:31.714700 28155 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39586 (request call id 427) took 3709 ms. Trace:
I20250905 20:00:31.715186 28155 rpcz_store.cc:276] 0905 20:00:28.004798 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:28.854255 (+849457us) service_pool.cc:225] Handling call
0905 20:00:28.854674 (+ 419us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:29.997143 (+1142469us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:31.714128 (+1716985us) spinlock_profiling.cc:243] Waited 863 ms on lock 0x7b5400073ad4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:31.714469 (+ 341us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1984932096}
I20250905 20:00:31.654541 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:31.655982 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:31.650131 28150 raft_consensus.cc:1238] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:31.669332 31388 leader_election.cc:290] T 36e45650becf4b5395e131ca159c32c4 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:31.667850 28002 pending_rounds.cc:85] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Aborting all ops after (but not including) 8
I20250905 20:00:31.677372 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:31.671908 28745 raft_consensus.cc:2466] T 986c4f475a8d46f39465775db70416d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 4.
I20250905 20:00:31.679729 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 20:00:31.680974 27974 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:31.681049 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 131974us
I20250905 20:00:31.659001 31403 raft_consensus.cc:738] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:31.689340 31458 raft_consensus.cc:513] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.690285 28006 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:31.735942 28002 pending_rounds.cc:107] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20250905 20:00:31.690575 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fba42101cd094077aa85e3f9186366e7" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:31.744143 28002 raft_consensus.cc:2887] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250905 20:00:31.691588 28750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13af7609cd04cf5b21745527b70074d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:31.693858 28158 raft_consensus.cc:2408] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20250905 20:00:31.693555 28004 raft_consensus.cc:2466] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 7.
W20250905 20:00:31.694203 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:31.695037 31315 leader_election.cc:290] T 50ac9970ddf042fda15f38b623aa7dd0 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:31.695886 28009 raft_consensus.cc:1238] T 36e45650becf4b5395e131ca159c32c4 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20250905 20:00:31.696125 31261 raft_consensus.cc:513] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:31.697660 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:31.747495 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:31.654354 31358 consensus_queue.cc:1035] T 7eae4eae1373464d85793607943a7c4d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.045s
I20250905 20:00:31.701804 31284 leader_election.cc:290] T ef7f42a26b64466a9d6b1eae6df0e614 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:31.702718 31275 raft_consensus.cc:491] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:31.705039 30946 raft_consensus.cc:491] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:31.670480 28752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:31.752322 28752 raft_consensus.cc:2408] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20250905 20:00:31.698895 28123 leader_election.cc:304] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:31.725560 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:31.752990 28152 raft_consensus.cc:2466] T 2e578d82f2b643a795b79abae746575e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:31.726116 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:31.726385 28007 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54156 (request call id 2219) took 3552 ms (client timeout 3278 ms). Trace:
I20250905 20:00:31.726979 31457 raft_consensus.cc:513] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.728098 28751 raft_consensus.cc:2391] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 8: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 20:00:31.728550 31426 raft_consensus.cc:513] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.729548 31355 raft_consensus.cc:513] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.714740 31357 raft_consensus.cc:513] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.731076 31220 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:31.756898 28001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05b2fd124328472aa3e3f62e0b327582" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:31.732358 31404 raft_consensus.cc:491] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:31.732553 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8afd77a8d80843aa805f971a2806ab86" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:31.734062 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:31.674801 31455 raft_consensus.cc:738] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:31.740098 28159 raft_consensus.cc:2466] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 7.
I20250905 20:00:31.733865 31463 leader_election.cc:290] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:31.738832 28003 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62ccdb07048e4762b1b31c36b14610f5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" is_pre_election: true
I20250905 20:00:31.715773 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:31.749768 28745 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:31.695108 28081 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:31.756426 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18c49d7ed5d64445afa2c6ee7a2e27d1" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:31.731299 31190 leader_election.cc:290] T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:31.762471 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
W20250905 20:00:31.753676 28007 rpcz_store.cc:269] 0905 20:00:28.174199 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:28.199569 (+ 25370us) service_pool.cc:225] Handling call
0905 20:00:31.681822 (+3482253us) spinlock_profiling.cc:243] Waited 360 ms on lock 0x7b54000541d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:31.726183 (+ 44361us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":828338560}
I20250905 20:00:31.766134 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:31.766657 28006 raft_consensus.cc:2466] T a8574207dbad48be8dcd599e9cae1ded P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 8.
W20250905 20:00:31.769850 27975 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:31.770682 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:31.771646 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 72879.2us
I20250905 20:00:31.775240 28747 raft_consensus.cc:2391] T 8afd77a8d80843aa805f971a2806ab86 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 8: Already voted for candidate 7e9c9f591f05464f9b097bf003d14f4a in this term.
I20250905 20:00:31.784363 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "761b89d47d5543d89dd47af039cfe7de" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:31.712733 28151 rpcz_store.cc:276] 0905 20:00:27.905443 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:28.829968 (+924525us) service_pool.cc:225] Handling call
0905 20:00:28.854359 (+ 24391us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:28.868804 (+ 14445us) raft_consensus.cc:1532] Early marking committed up to index 4
0905 20:00:29.107143 (+238339us) log.cc:844] Serialized 14 byte log entry
0905 20:00:29.273209 (+166066us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:29.273289 (+ 80us) raft_consensus.cc:1643] Marking committed up to 4
0905 20:00:29.273392 (+ 103us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:31.089276 (+1815884us) spinlock_profiling.cc:243] Waited 5.88 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49d85891e 00007fe49d857c66 00007fe49d857888 00007fe49d8612c1 00007fe49d861279 00007fe49d861209
0905 20:00:31.394811 (+305535us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b18001c0b40. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49ac723d6 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe4a3963e41 00007fe4a3970791 00007fe49aca43e2 00007fe49aca67b1
0905 20:00:31.711882 (+317071us) spinlock_profiling.cc:243] Waited 8.32 ms on lock 0x7b540037f0d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:31.711928 (+ 46us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:31.712153 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14135,"prepare.queue_time_us":483446,"raft.queue_time_us":7147,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":35568,"spinlock_wait_cycles":104803584,"thread_start_us":389611,"threads_started":2,"wal-append.queue_time_us":450261}
I20250905 20:00:31.823062 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c1867b70e8143a5b3d46d269c9c139c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:31.826161 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3efc6666ba5411c9b50c9d6e34089ad" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:31.827049 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:31.827250 28001 raft_consensus.cc:2466] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 8.
I20250905 20:00:31.827906 28151 raft_consensus.cc:2391] T 1f65256308814d3a9a4a01db1b910611 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 6: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:31.828963 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:31.824352 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 49897.5us
I20250905 20:00:31.774567 28080 raft_consensus.cc:3058] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:31.829802 28151 raft_consensus.cc:2391] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 9: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:31.832747 28751 raft_consensus.cc:2408] T 761b89d47d5543d89dd47af039cfe7de P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
W20250905 20:00:31.830371 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:31.827095 28158 raft_consensus.cc:2466] T a3efc6666ba5411c9b50c9d6e34089ad P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 7.
I20250905 20:00:31.841890 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:31.846963 28050 connection.cc:232] RPC call timeout handler was delayed by -1.0173110099999576s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1665s for a response
I20250905 20:00:31.850265 31284 consensus_queue.cc:1046] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250905 20:00:31.850736 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:31.850848 31403 consensus_queue.cc:260] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.851621 28082 raft_consensus.cc:2466] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 9.
I20250905 20:00:31.851797 31284 raft_consensus.cc:3053] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Stepping down as leader of term 4
I20250905 20:00:31.852176 31403 raft_consensus.cc:3058] T 1f65256308814d3a9a4a01db1b910611 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:31.829200 31321 consensus_queue.cc:1035] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.025s
I20250905 20:00:31.852362 31284 raft_consensus.cc:738] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
W20250905 20:00:31.854286 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 101761us
I20250905 20:00:31.855079 31261 leader_election.cc:290] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
W20250905 20:00:31.855706 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:31.856299 31467 raft_consensus.cc:491] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:31.856555 27973 leader_election.cc:304] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:31.857077 31467 raft_consensus.cc:513] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.858688 31332 raft_consensus.cc:491] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:31.859035 31355 leader_election.cc:290] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:31.859373 31332 raft_consensus.cc:513] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.859767 31355 consensus_queue.cc:1035] T ff701570f51b481a81cb71ede12a494f P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 20:00:31.835928 28153 raft_consensus.cc:2466] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
I20250905 20:00:31.839862 28002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1654) took 2865 ms. Trace:
I20250905 20:00:31.859568 30946 raft_consensus.cc:513] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.862085 28002 rpcz_store.cc:276] 0905 20:00:28.974383 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:30.763958 (+1789575us) service_pool.cc:225] Handling call
0905 20:00:30.812078 (+ 48120us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:31.776300 (+964222us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:31.839794 (+ 63494us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 20:00:31.863792 28154 raft_consensus.cc:2466] T ff701570f51b481a81cb71ede12a494f P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 7.
I20250905 20:00:31.771098 31458 leader_election.cc:290] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:31.849931 27974 leader_election.cc:304] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:31.756758 31427 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
W20250905 20:00:31.862421 27975 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:31.866230 28749 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:31.865643 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:31.878270 28151 raft_consensus.cc:2466] T 1d1a8f2db2604a879529dda40b457a4a P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:31.849951 28050 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters: cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:31.774639 31402 raft_consensus.cc:491] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
W20250905 20:00:31.906881 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 70523.1us
I20250905 20:00:31.910773 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:31.911655 28077 raft_consensus.cc:3058] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:31.912897 31332 leader_election.cc:290] T 6c423052d6454829bccfe7c8bc1bda5c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:31.913713 31298 raft_consensus.cc:491] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:31.914554 31467 leader_election.cc:290] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:31.914455 31298 raft_consensus.cc:513] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:31.916837 31428 log.cc:927] Time spent T 6dbd54765afc49cb862b267d6663000c P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 1.383s user 0.003s sys 0.002s
I20250905 20:00:31.919162 28082 raft_consensus.cc:1238] T 2bd6bb972f14472a89ccf99de880efc2 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:31.919518 31457 leader_election.cc:290] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:31.775264 31322 consensus_queue.cc:1046] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.109s
I20250905 20:00:31.920171 30946 leader_election.cc:290] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:31.920495 31322 consensus_queue.cc:1230] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer 7e9c9f591f05464f9b097bf003d14f4a log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20250905 20:00:31.920907 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 64490.9us
I20250905 20:00:31.921207 31217 raft_consensus.cc:2802] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election won for term 6
W20250905 20:00:31.921545 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 71775.2us
I20250905 20:00:31.921813 31217 raft_consensus.cc:695] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [term 6 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:31.922446 27973 leader_election.cc:304] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:31.909409 28003 raft_consensus.cc:2466] T 62ccdb07048e4762b1b31c36b14610f5 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 6.
I20250905 20:00:31.927525 28150 raft_consensus.cc:1238] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Rejecting Update request from peer e4413ae5e0b64b52a6f360ac580239b9 for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:31.859169 31426 leader_election.cc:290] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:31.883975 31427 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:31.880403 31358 raft_consensus.cc:2747] T 05b2fd124328472aa3e3f62e0b327582 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250905 20:00:31.939760 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:31.940654 28083 raft_consensus.cc:2466] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 7.
I20250905 20:00:31.933769 28049 leader_election.cc:304] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:31.938892 28082 raft_consensus.cc:1238] T 761b89d47d5543d89dd47af039cfe7de P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Rejecting Update request from peer 7e9c9f591f05464f9b097bf003d14f4a for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20250905 20:00:31.941772 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ed920d4c0144d62acf66f595bbf34fd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:31.941936 31177 raft_consensus.cc:3053] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 LEADER]: Stepping down as leader of term 6
I20250905 20:00:31.942497 31177 raft_consensus.cc:738] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 26b2f874b8e54f08b01a9ed00d5cce05, State: Running, Role: LEADER
I20250905 20:00:31.942644 28049 leader_election.cc:304] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:31.775995 31275 raft_consensus.cc:513] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.943935 28049 leader_election.cc:304] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a, cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:31.943835 31471 raft_consensus.cc:2802] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 20:00:31.870106 31388 raft_consensus.cc:2802] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 20:00:31.944823 31471 raft_consensus.cc:491] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:31.945278 31471 raft_consensus.cc:3058] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:31.945353 31457 raft_consensus.cc:2802] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 20:00:31.945847 31457 raft_consensus.cc:491] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:31.946200 31457 raft_consensus.cc:3058] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:31.835654 31404 raft_consensus.cc:513] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:31.943037 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:31.948352 31404 leader_election.cc:290] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:31.949011 31404 raft_consensus.cc:2747] T fc6a8d5d253643cea8582bef1c4173d6 P e4413ae5e0b64b52a6f360ac580239b9 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250905 20:00:31.950760 31298 leader_election.cc:290] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:31.952339 27973 leader_election.cc:304] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:31.958684 28158 raft_consensus.cc:1273] T 0e4ec392d0d54040b0b2162eb0601c0d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20250905 20:00:31.958989 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:31.961454 31284 consensus_queue.cc:260] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:31.944254 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "986c4f475a8d46f39465775db70416d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:31.962750 31276 raft_consensus.cc:2747] T 5f31007f29044bd1817fcf19ba904ad3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250905 20:00:31.964633 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:31.968230 28009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54164 (request call id 1639) took 3316 ms. Trace:
I20250905 20:00:31.971341 28156 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2348) took 3708 ms. Trace:
I20250905 20:00:31.982935 28009 rpcz_store.cc:276] 0905 20:00:28.651684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:29.271655 (+619971us) service_pool.cc:225] Handling call
0905 20:00:31.695267 (+2423612us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:31.761803 (+ 66536us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:31.967920 (+206117us) spinlock_profiling.cc:243] Waited 2.32 s on lock 0x7b54000541d0. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1886d 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:31.968136 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5328587520}
I20250905 20:00:31.983177 28156 rpcz_store.cc:276] 0905 20:00:28.262791 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:28.962651 (+699860us) service_pool.cc:225] Handling call
0905 20:00:28.975714 (+ 13063us) raft_consensus.cc:1469] Updating replica for 1 ops
0905 20:00:28.976186 (+ 472us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:28.976229 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0905 20:00:29.484567 (+508338us) spinlock_profiling.cc:243] Waited 6.17 ms on lock 0x7b2c0002df18. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49c5bc9b1 00007fe49c5cbfe2 00007fe49dc55021 00007fe49dc193fa 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32
0905 20:00:29.525669 (+ 41102us) log.cc:844] Serialized 26 byte log entry
0905 20:00:29.711735 (+186066us) raft_consensus.cc:1643] Marking committed up to 8
0905 20:00:29.957739 (+246004us) log.cc:844] Serialized 14 byte log entry
0905 20:00:29.957969 (+ 230us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:29.958053 (+ 84us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 20:00:31.915871 (+1957818us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b440002304c. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49abb3ba5 00007fe49abb45c2 00007fe49d869231 00007fe49d864b5a 00007fe49d85891e 00007fe49d857c66 00007fe49d857888 00007fe49d8612c1 00007fe49d861279 00007fe49d861209
0905 20:00:31.970808 (+ 54937us) raft_consensus.cc:1683] finished
0905 20:00:31.970832 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:31.971150 (+ 318us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":462477,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":38860,"raft.queue_time_us":269908,"raft.run_cpu_time_us":486,"raft.run_wall_time_us":104158,"spinlock_wait_cycles":45914752,"thread_start_us":145288,"threads_started":1,"wal-append.queue_time_us":525851}
I20250905 20:00:31.970863 31388 raft_consensus.cc:491] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:31.985970 31357 leader_election.cc:290] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:31.972723 28083 raft_consensus.cc:3058] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:31.923233 31217 consensus_queue.cc:237] T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } attrs { replace: true } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:31.993249 28076 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1727) took 2380 ms (client timeout 1500 ms). Trace:
W20250905 20:00:31.993798 28076 rpcz_store.cc:269] 0905 20:00:29.612690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:31.992868 (+2380178us) service_pool.cc:212] Skipping call since client already timed out
0905 20:00:31.993048 (+ 180us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250905 20:00:31.961211 28084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:31.986178 31388 raft_consensus.cc:3058] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 2 FOLLOWER]: Advancing to term 3
I20250905 20:00:31.998826 30946 consensus_queue.cc:1035] T 8a8d8393352b4214a20c1683852d4fdd P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20250905 20:00:32.003082 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 90719.2us
I20250905 20:00:32.004133 28076 raft_consensus.cc:1273] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20250905 20:00:32.005172 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:32.003512 31471 raft_consensus.cc:513] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:31.884424 31220 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:32.005640 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:32.005807 28076 raft_consensus.cc:3058] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Advancing to term 10
W20250905 20:00:31.971684 28085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59550 (request call id 1721) took 2697 ms (client timeout 1636 ms). Trace:
I20250905 20:00:32.012568 28079 raft_consensus.cc:2466] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
W20250905 20:00:32.005970 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 59557.6us
W20250905 20:00:31.962643 27974 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 112835us
I20250905 20:00:32.005766 28744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40926 (request call id 559) took 1771 ms. Trace:
I20250905 20:00:31.972445 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:31.929458 28123 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 90720.9us
W20250905 20:00:32.014336 28085 rpcz_store.cc:269] 0905 20:00:29.273918 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:29.274866 (+ 948us) service_pool.cc:225] Handling call
0905 20:00:31.971437 (+2696571us) spinlock_profiling.cc:243] Waited 1.6 s on lock 0x7b540019f5d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc1d2aa 00007fe4a2b7d5d7 00007fe49c2050c4 00007fe49c205052 00007fe49c204f81 00007fe49c204efc 00007fe49c2041b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:31.971609 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3685889280}
I20250905 20:00:32.017987 28744 rpcz_store.cc:276] 0905 20:00:30.234031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:30.234246 (+ 215us) service_pool.cc:225] Handling call
0905 20:00:31.096250 (+862004us) spinlock_profiling.cc:243] Waited 5.93 ms on lock 0x7b5800304598. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49aae5531 00007fe49ac66ea1 00007fe49ac66d52 00007fe4a397cc49 00007fe4a2b08010 00007fe4a2bceebb 00007fe4a2bcef41 00007fe4a2b91f60 00007fe4a2b7d1a9 00007fe4a2b7cc52 00007fe49c202b44
0905 20:00:31.096529 (+ 279us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:31.522255 (+425726us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:32.005326 (+483071us) spinlock_profiling.cc:243] Waited 18.1 ms on lock 0x7b540048c6d4. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:32.005600 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":55317760}
W20250905 20:00:32.020989 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:32.026887 28077 raft_consensus.cc:2466] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 8.
I20250905 20:00:32.019428 28081 raft_consensus.cc:2466] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 9.
I20250905 20:00:32.029237 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a002b516fab044b998f136c341a88b74" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:31.946141 31275 leader_election.cc:290] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:32.024483 28158 raft_consensus.cc:2466] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 8.
I20250905 20:00:32.035317 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:32.035524 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:32.036495 28751 raft_consensus.cc:2391] T fc7dc431504743d79dbdc8eea83f9dcb P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in current term 6: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:32.041661 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dbd54765afc49cb862b267d6663000c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:32.043946 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:32.045271 28085 raft_consensus.cc:2466] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 2.
I20250905 20:00:31.943688 31177 consensus_queue.cc:260] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:32.052999 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:32.057684 28158 raft_consensus.cc:2466] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
I20250905 20:00:32.054286 27974 leader_election.cc:304] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:32.058615 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:32.059904 28156 raft_consensus.cc:2391] T e707b0f561934674947fb463ce626a07 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 8: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:32.065537 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:32.066789 28156 raft_consensus.cc:2391] T a8574207dbad48be8dcd599e9cae1ded P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 9: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:32.046768 31276 raft_consensus.cc:2802] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 20:00:32.043921 31358 raft_consensus.cc:2747] T 761b89d47d5543d89dd47af039cfe7de P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250905 20:00:32.032773 31402 raft_consensus.cc:513] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } }
I20250905 20:00:32.046875 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:32.071596 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a9976ebf1c1415a85592e195bad9f36" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
W20250905 20:00:32.073532 28050 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:32.074448 28123 leader_election.cc:304] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:32.078114 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:32.079079 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:32.082924 31453 raft_consensus.cc:2747] T fc7dc431504743d79dbdc8eea83f9dcb P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250905 20:00:32.087983 28154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:32.088929 28749 raft_consensus.cc:1238] T 5ed920d4c0144d62acf66f595bbf34fd P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
W20250905 20:00:32.083561 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 131354us
I20250905 20:00:32.104641 28076 raft_consensus.cc:2466] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 10.
I20250905 20:00:32.096125 28153 raft_consensus.cc:2391] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 6: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:32.071391 28085 raft_consensus.cc:1238] T e802ce1da002477b8896acf7bf7c84a7 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 4. Current term is 5. Ops: []
I20250905 20:00:32.124251 28083 raft_consensus.cc:2466] T 5ed920d4c0144d62acf66f595bbf34fd P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 8.
W20250905 20:00:32.129567 31084 consensus_meta.cc:287] T b12e279a4fab4ea3a15dadc55d224ddb P cd764bfeebac4cc9ad851cd4a6a7d3b0: Time spent flushing consensus metadata: real 0.600s user 0.009s sys 0.006s
I20250905 20:00:32.130831 31402 leader_election.cc:290] T bbd67b7a3c3244838ec2d727f746cc0b P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:32.110085 31261 raft_consensus.cc:2802] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250905 20:00:32.134944 31261 raft_consensus.cc:491] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Starting leader election (detected failure of leader cd764bfeebac4cc9ad851cd4a6a7d3b0)
I20250905 20:00:32.135033 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:32.135459 31261 raft_consensus.cc:3058] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Advancing to term 9
I20250905 20:00:32.143250 28078 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59550 (request call id 1729) took 2450 ms. Trace:
I20250905 20:00:32.143651 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d06337b10d154efbbaaf3c152d01f6dd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:32.144074 28078 rpcz_store.cc:276] 0905 20:00:29.692662 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:29.692867 (+ 205us) service_pool.cc:225] Handling call
0905 20:00:29.719181 (+ 26314us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:30.444731 (+725550us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 20:00:30.444773 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0905 20:00:30.444815 (+ 42us) raft_consensus.cc:1643] Marking committed up to 3
0905 20:00:30.444857 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:32.142532 (+1697675us) spinlock_profiling.cc:243] Waited 725 ms on lock 0x7b540021fa54. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:32.142627 (+ 95us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 20:00:32.142969 (+ 342us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1667853184}
I20250905 20:00:32.145484 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:32.146893 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:32.147907 28744 raft_consensus.cc:2391] T 8a8d8393352b4214a20c1683852d4fdd P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 9: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:32.149734 28152 raft_consensus.cc:2391] T b12e279a4fab4ea3a15dadc55d224ddb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 10: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:32.120410 28743 raft_consensus.cc:2466] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 2.
I20250905 20:00:32.150874 31455 consensus_queue.cc:260] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:32.151830 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c423052d6454829bccfe7c8bc1bda5c" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:32.151995 31455 raft_consensus.cc:3058] T 0860997b3256400084f5884190bbe6e2 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:32.143260 28082 raft_consensus.cc:1238] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 7. Current term is 8. Ops: []
I20250905 20:00:32.147341 28079 raft_consensus.cc:1238] T 5f31007f29044bd1817fcf19ba904ad3 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 LEADER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20250905 20:00:32.146142 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:32.170555 28078 raft_consensus.cc:2466] T 24cf07baf43247b6bf63063106b79836 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:32.175484 28049 leader_election.cc:304] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:32.177235 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:32.062922 31275 consensus_queue.cc:1046] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.091s
I20250905 20:00:32.187055 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff701570f51b481a81cb71ede12a494f" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:32.181974 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:32.188524 28085 raft_consensus.cc:2466] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
I20250905 20:00:32.182878 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:32.086091 31298 raft_consensus.cc:491] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 26b2f874b8e54f08b01a9ed00d5cce05)
I20250905 20:00:32.192150 28080 raft_consensus.cc:2466] T fba42101cd094077aa85e3f9186366e7 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 5.
I20250905 20:00:32.181249 28153 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24cf07baf43247b6bf63063106b79836" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:32.198915 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:32.202953 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:32.209085 28158 raft_consensus.cc:1273] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250905 20:00:32.216192 30978 consensus_queue.cc:1035] T 18ed409dabfc45b1843e7d8a316a047d P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20250905 20:00:32.216612 28743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:32.217398 28743 raft_consensus.cc:2466] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 9.
I20250905 20:00:32.217458 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:32.218144 28082 raft_consensus.cc:3058] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Advancing to term 11
I20250905 20:00:32.218559 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8574207dbad48be8dcd599e9cae1ded" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:32.192363 31471 leader_election.cc:290] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 election: Requested vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:32.220208 28749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "361ffd5c9a994e0397e6748827466d3d" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:32.220954 28749 raft_consensus.cc:3058] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:32.200490 31196 raft_consensus.cc:2747] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250905 20:00:32.225329 28077 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:32.202049 31220 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:32.228951 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:32.236531 28079 raft_consensus.cc:3058] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Advancing to term 9
I20250905 20:00:32.236465 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e578d82f2b643a795b79abae746575e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:32.215245 28049 leader_election.cc:304] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:32.241174 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:32.241413 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:32.241925 28076 raft_consensus.cc:3058] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:32.240551 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "456d9fe0918b423abeab5c39372bcbe9" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:32.191496 31298 raft_consensus.cc:513] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:32.246788 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:32.247386 31341 raft_consensus.cc:491] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:32.252027 31341 raft_consensus.cc:513] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:32.256331 31276 raft_consensus.cc:491] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:32.256909 31276 raft_consensus.cc:3058] T 986c4f475a8d46f39465775db70416d3 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:32.259586 31298 leader_election.cc:290] T 8e0ac2dfe59d423189dd8d72eee98a42 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:32.260221 28151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8a1b71a731e455da7990629d2c94801" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:32.261116 28151 raft_consensus.cc:2391] T b8a1b71a731e455da7990629d2c94801 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 7: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:32.264729 28158 raft_consensus.cc:2466] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 8.
I20250905 20:00:32.265465 31177 raft_consensus.cc:3058] T 0c1867b70e8143a5b3d46d269c9c139c P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Advancing to term 7
I20250905 20:00:32.265975 31341 leader_election.cc:290] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193)
I20250905 20:00:32.270069 28749 raft_consensus.cc:2466] T 361ffd5c9a994e0397e6748827466d3d P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 8.
W20250905 20:00:32.280405 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:32.292742 28049 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.132:44299, user_credentials={real_user=slave}} blocked reactor thread for 91102.6us
I20250905 20:00:32.293973 28049 leader_election.cc:304] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 20:00:32.295794 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:32.297892 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a8d8393352b4214a20c1683852d4fdd" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:32.289454 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 47097.6us
I20250905 20:00:32.204577 31453 consensus_queue.cc:1046] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20250905 20:00:32.307654 28748 raft_consensus.cc:2408] T 18ed409dabfc45b1843e7d8a316a047d P 7e9c9f591f05464f9b097bf003d14f4a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250905 20:00:32.307549 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b12e279a4fab4ea3a15dadc55d224ddb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:32.308317 28080 raft_consensus.cc:2374] T b12e279a4fab4ea3a15dadc55d224ddb P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 10. Re-sending same reply.
I20250905 20:00:32.310024 28077 raft_consensus.cc:2391] T 8a8d8393352b4214a20c1683852d4fdd P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 in current term 9: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:32.241042 31358 raft_consensus.cc:491] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:32.313263 27975 leader_election.cc:304] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:32.319792 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:32.325582 28082 raft_consensus.cc:2466] T 6db33851e9914ca58560e8902701dca3 P e4413ae5e0b64b52a6f360ac580239b9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 11.
I20250905 20:00:32.332057 28079 raft_consensus.cc:2466] T a8574207dbad48be8dcd599e9cae1ded P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 9.
I20250905 20:00:32.340471 31366 consensus_queue.cc:1035] T c0889204295e4daa99d81e5b2a45ac1c P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20250905 20:00:32.302856 31471 raft_consensus.cc:2802] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250905 20:00:32.341818 31471 raft_consensus.cc:491] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:32.302697 31196 raft_consensus.cc:2802] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 20:00:32.342651 31196 raft_consensus.cc:491] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:32.343019 31196 raft_consensus.cc:3058] T f2de38aacf1b4dccaeee999400fb6d75 P e4413ae5e0b64b52a6f360ac580239b9 [term 2 FOLLOWER]: Advancing to term 3
I20250905 20:00:32.349226 31084 raft_consensus.cc:3053] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 LEADER]: Stepping down as leader of term 7
I20250905 20:00:32.349704 31084 raft_consensus.cc:738] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:32.351336 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10dd790c7b5c4cc5978c3545b5a06868" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:32.354866 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 43635.4us
I20250905 20:00:32.349049 28080 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:32.342273 31471 raft_consensus.cc:3058] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Advancing to term 10
I20250905 20:00:32.372654 28049 leader_election.cc:304] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9; no voters:
I20250905 20:00:32.377210 28079 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:32.388482 28050 leader_election.cc:304] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0
W20250905 20:00:32.396057 31496 log.cc:927] Time spent T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20250905 20:00:32.401232 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:32.394006 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:32.407828 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e45650becf4b5395e131ca159c32c4" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:32.404208 28158 raft_consensus.cc:2466] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 8.
I20250905 20:00:32.412598 28082 raft_consensus.cc:2466] T 36e45650becf4b5395e131ca159c32c4 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 8.
I20250905 20:00:32.415027 31217 raft_consensus.cc:2802] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader election won for term 8
I20250905 20:00:32.415539 28744 raft_consensus.cc:1238] T 9a9976ebf1c1415a85592e195bad9f36 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 7. Current term is 8. Ops: []
I20250905 20:00:32.415820 28751 raft_consensus.cc:1273] T 4f2279535926437ba84ae91f2b83b3d3 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Refusing update from remote peer 26b2f874b8e54f08b01a9ed00d5cce05: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20250905 20:00:32.426414 28076 raft_consensus.cc:2466] T b8a1b71a731e455da7990629d2c94801 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
W20250905 20:00:32.428869 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 115709us
W20250905 20:00:32.430418 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 111163us
I20250905 20:00:32.434324 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50ac9970ddf042fda15f38b623aa7dd0" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:32.437124 28158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:32.437517 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:32.444204 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c21c3755f90405db03f98780efdb0b5" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
W20250905 20:00:32.446113 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:32.421733 31322 raft_consensus.cc:2747] T fd2b679c53994263846f51ceec97a150 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20250905 20:00:32.454650 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:32.455973 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:32.459800 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:32.450630 31217 raft_consensus.cc:695] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [term 8 LEADER]: Becoming Leader. State: Replica: e4413ae5e0b64b52a6f360ac580239b9, State: Running, Role: LEADER
I20250905 20:00:32.460700 28156 raft_consensus.cc:2466] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4413ae5e0b64b52a6f360ac580239b9 in term 5.
I20250905 20:00:32.462177 28076 raft_consensus.cc:2466] T 5c21c3755f90405db03f98780efdb0b5 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
W20250905 20:00:32.492247 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:32.458103 31457 consensus_meta.cc:287] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9: Time spent flushing consensus metadata: real 0.511s user 0.012s sys 0.002s
I20250905 20:00:32.456624 31463 consensus_queue.cc:1046] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.003s
I20250905 20:00:32.402864 31358 raft_consensus.cc:513] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:32.473896 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d1a8f2db2604a879529dda40b457a4a" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:32.475045 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4ec392d0d54040b0b2162eb0601c0d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 10 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:32.507391 28078 raft_consensus.cc:2408] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20250905 20:00:32.475711 27976 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:32.476882 31284 raft_consensus.cc:3058] T fba42101cd094077aa85e3f9186366e7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 4 FOLLOWER]: Advancing to term 5
I20250905 20:00:32.477546 31449 raft_consensus.cc:2747] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250905 20:00:32.468072 28085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f356a2522e646889c023d4b3d26f9cc" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:32.507512 27975 leader_election.cc:304] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:32.494940 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10df9d019f2e4e208f4e23d50f551ad6" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:32.502635 28076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:32.511013 28076 raft_consensus.cc:2466] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 5.
I20250905 20:00:32.447582 31341 consensus_queue.cc:1046] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.039s
I20250905 20:00:32.511798 28156 raft_consensus.cc:2408] T 10df9d019f2e4e208f4e23d50f551ad6 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20250905 20:00:32.445705 31388 raft_consensus.cc:513] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
W20250905 20:00:32.517390 27973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 53325.2us
I20250905 20:00:32.459319 31084 consensus_queue.cc:260] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:32.453678 31479 consensus_queue.cc:1035] T 4f2279535926437ba84ae91f2b83b3d3 P 26b2f874b8e54f08b01a9ed00d5cce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.045s
I20250905 20:00:32.530565 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca47360fb0f4be6a95a798d0a6d25b8" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 12 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:32.536628 31358 leader_election.cc:290] T 9f7f6e4e3a754dcda90928621168fbe7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:32.539115 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db33851e9914ca58560e8902701dca3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:32.539933 27973 leader_election.cc:304] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:32.540410 28152 raft_consensus.cc:2391] T 6db33851e9914ca58560e8902701dca3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 11: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:32.541838 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbd67b7a3c3244838ec2d727f746cc0b" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:32.542898 31358 raft_consensus.cc:2747] T 0c1867b70e8143a5b3d46d269c9c139c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250905 20:00:32.541985 27973 leader_election.cc:304] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters: 7e9c9f591f05464f9b097bf003d14f4a, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:32.547549 31322 raft_consensus.cc:2747] T 2bd6bb972f14472a89ccf99de880efc2 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250905 20:00:32.552636 27973 leader_election.cc:304] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd764bfeebac4cc9ad851cd4a6a7d3b0, e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05
I20250905 20:00:32.555757 31298 raft_consensus.cc:3053] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 LEADER]: Stepping down as leader of term 7
I20250905 20:00:32.556561 31388 leader_election.cc:290] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [CANDIDATE]: Term 3 election: Requested vote from peers e4413ae5e0b64b52a6f360ac580239b9 (127.26.58.130:34193), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:32.556908 31298 raft_consensus.cc:738] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 7e9c9f591f05464f9b097bf003d14f4a, State: Running, Role: LEADER
I20250905 20:00:32.563849 28158 raft_consensus.cc:2391] T f2de38aacf1b4dccaeee999400fb6d75 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4413ae5e0b64b52a6f360ac580239b9 in current term 3: Already voted for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in this term.
I20250905 20:00:32.568795 28748 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:32.569801 28748 raft_consensus.cc:3058] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 2 FOLLOWER]: Advancing to term 3
I20250905 20:00:32.570003 28751 raft_consensus.cc:1273] T fc6a8d5d253643cea8582bef1c4173d6 P 7e9c9f591f05464f9b097bf003d14f4a [term 12 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20250905 20:00:32.571838 27975 leader_election.cc:304] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 20:00:32.573895 28155 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.025s sys 0.095s
W20250905 20:00:32.574478 28155 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.573s user 0.026s sys 0.095s
I20250905 20:00:32.574781 28078 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f65256308814d3a9a4a01db1b910611" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 20:00:32.579372 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 201354us
I20250905 20:00:32.583832 28082 raft_consensus.cc:2466] T 7ca47360fb0f4be6a95a798d0a6d25b8 P e4413ae5e0b64b52a6f360ac580239b9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 11.
I20250905 20:00:32.559808 31402 raft_consensus.cc:2802] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250905 20:00:32.587977 27975 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 59055.1us
I20250905 20:00:32.585188 28078 raft_consensus.cc:3058] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:32.585906 28744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f7f6e4e3a754dcda90928621168fbe7" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a" is_pre_election: true
I20250905 20:00:32.592402 27975 leader_election.cc:304] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:32.558166 31298 consensus_queue.cc:260] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:32.514319 31457 raft_consensus.cc:513] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:32.591511 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:32.593847 27975 leader_election.cc:304] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
I20250905 20:00:32.606370 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aace14abd237420c8211ce611ea09f0e" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:32.608095 28152 raft_consensus.cc:2466] T aace14abd237420c8211ce611ea09f0e P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 7.
I20250905 20:00:32.613032 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7eae4eae1373464d85793607943a7c4d" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9" is_pre_election: true
I20250905 20:00:32.616732 28082 raft_consensus.cc:2466] T 7eae4eae1373464d85793607943a7c4d P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 7.
I20250905 20:00:32.618592 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef7f42a26b64466a9d6b1eae6df0e614" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:32.589284 30946 raft_consensus.cc:491] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:32.612813 27975 leader_election.cc:304] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, cd764bfeebac4cc9ad851cd4a6a7d3b0; no voters:
W20250905 20:00:32.593986 31485 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.676s user 0.017s sys 0.098s
I20250905 20:00:32.565668 31471 raft_consensus.cc:513] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:32.632372 28717 leader_election.cc:304] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:32.596735 31402 raft_consensus.cc:491] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:32.629585 28084 raft_consensus.cc:2391] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e9c9f591f05464f9b097bf003d14f4a in current term 8: Already voted for candidate e4413ae5e0b64b52a6f360ac580239b9 in this term.
I20250905 20:00:32.596036 31084 raft_consensus.cc:3058] T 5ed920d4c0144d62acf66f595bbf34fd P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:32.598105 30978 raft_consensus.cc:2802] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:32.601058 31282 raft_consensus.cc:2802] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 20:00:32.629047 28156 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0860997b3256400084f5884190bbe6e2" candidate_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:32.619654 28155 raft_consensus.cc:2466] T ef7f42a26b64466a9d6b1eae6df0e614 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 5.
I20250905 20:00:32.615549 31454 raft_consensus.cc:2802] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250905 20:00:32.641391 28152 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc7dc431504743d79dbdc8eea83f9dcb" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
W20250905 20:00:32.631772 31485 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.714s user 0.017s sys 0.099s
I20250905 20:00:32.629719 31358 raft_consensus.cc:2802] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader election won for term 5
I20250905 20:00:32.676517 28152 raft_consensus.cc:2466] T fc7dc431504743d79dbdc8eea83f9dcb P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6.
I20250905 20:00:32.710793 28080 raft_consensus.cc:2366] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 9. Current term is 10.
W20250905 20:00:32.711057 28718 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 20:00:32.700592 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:32.637805 31463 raft_consensus.cc:491] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 20:00:32.715689 28715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.130:34193, user_credentials={real_user=slave}} blocked reactor thread for 168594us
W20250905 20:00:32.716066 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 49854.9us
I20250905 20:00:32.612438 31322 consensus_queue.cc:1035] T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.059s
I20250905 20:00:32.638226 31402 raft_consensus.cc:3058] T 50ac9970ddf042fda15f38b623aa7dd0 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:32.681972 31454 raft_consensus.cc:491] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:32.723089 28085 raft_consensus.cc:1238] T 0c1867b70e8143a5b3d46d269c9c139c P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 6. Current term is 7. Ops: []
I20250905 20:00:32.640688 30978 raft_consensus.cc:491] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:32.737566 30978 raft_consensus.cc:3058] T 2e578d82f2b643a795b79abae746575e P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
W20250905 20:00:32.716281 28122 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:32.647022 28156 raft_consensus.cc:2374] T 0860997b3256400084f5884190bbe6e2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cd764bfeebac4cc9ad851cd4a6a7d3b0 in term 7. Re-sending same reply.
W20250905 20:00:32.740775 28123 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:32.731144 28050 leader_election.cc:304] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters: 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:32.635391 30946 raft_consensus.cc:513] T e802ce1da002477b8896acf7bf7c84a7 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:32.597179 31217 consensus_queue.cc:237] T 361ffd5c9a994e0397e6748827466d3d P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:32.675340 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e707b0f561934674947fb463ce626a07" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
I20250905 20:00:32.741132 28155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0889204295e4daa99d81e5b2a45ac1c" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:32.743763 28050 leader_election.cc:304] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:32.745771 28083 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4414c08fe910403b80af2d558aac36b2" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 20:00:32.643923 31339 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.602s user 0.006s sys 0.070s
W20250905 20:00:32.715272 31261 consensus_meta.cc:287] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05: Time spent flushing consensus metadata: real 0.579s user 0.012s sys 0.002s
I20250905 20:00:32.749060 28083 raft_consensus.cc:3058] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:32.718819 31341 raft_consensus.cc:2802] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250905 20:00:32.719237 31433 raft_consensus.cc:3058] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Advancing to term 6
W20250905 20:00:32.750535 28715 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:32.752592 28152 raft_consensus.cc:1273] T 2f356a2522e646889c023d4b3d26f9cc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 5 FOLLOWER]: Refusing update from remote peer 7e9c9f591f05464f9b097bf003d14f4a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250905 20:00:32.755218 28082 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2de38aacf1b4dccaeee999400fb6d75" candidate_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4413ae5e0b64b52a6f360ac580239b9"
W20250905 20:00:32.805917 28718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.129:38421, user_credentials={real_user=slave}} blocked reactor thread for 63100.3us
I20250905 20:00:32.732470 31453 raft_consensus.cc:2802] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:32.807361 28715 leader_election.cc:304] T 6dbd54765afc49cb862b267d6663000c P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e9c9f591f05464f9b097bf003d14f4a; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9
I20250905 20:00:32.763454 31457 leader_election.cc:290] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:32.768009 31498 log.cc:927] Time spent T fc6a8d5d253643cea8582bef1c4173d6 P cd764bfeebac4cc9ad851cd4a6a7d3b0: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
W20250905 20:00:32.768543 28125 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:32.787135 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ed409dabfc45b1843e7d8a316a047d" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:32.816708 28150 raft_consensus.cc:2466] T 18ed409dabfc45b1843e7d8a316a047d P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c9f591f05464f9b097bf003d14f4a in term 6.
I20250905 20:00:32.726907 31298 raft_consensus.cc:3058] T e707b0f561934674947fb463ce626a07 P 7e9c9f591f05464f9b097bf003d14f4a [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:32.807451 28751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
I20250905 20:00:32.818173 28751 raft_consensus.cc:3058] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [term 9 FOLLOWER]: Advancing to term 10
I20250905 20:00:32.757824 31220 raft_consensus.cc:2802] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:32.740499 31454 raft_consensus.cc:3058] T ff701570f51b481a81cb71ede12a494f P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Advancing to term 8
W20250905 20:00:32.810947 28717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 70866.8us
W20250905 20:00:32.812821 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 69168.9us
I20250905 20:00:32.825924 28050 leader_election.cc:304] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4413ae5e0b64b52a6f360ac580239b9; no voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a
I20250905 20:00:32.646046 31282 raft_consensus.cc:491] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:32.810312 31261 raft_consensus.cc:513] T 1b18b7149ff3439a8dde09149cf336b3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:32.812103 31453 raft_consensus.cc:491] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader e4413ae5e0b64b52a6f360ac580239b9)
I20250905 20:00:32.824705 31171 raft_consensus.cc:3053] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 LEADER]: Stepping down as leader of term 7
I20250905 20:00:32.824743 31220 raft_consensus.cc:491] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7e9c9f591f05464f9b097bf003d14f4a)
I20250905 20:00:32.824743 28747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "7e9c9f591f05464f9b097bf003d14f4a"
W20250905 20:00:32.858741 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:32.805568 31341 raft_consensus.cc:491] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:32.835234 28152 raft_consensus.cc:1238] T 4414c08fe910403b80af2d558aac36b2 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Rejecting Update request from peer cd764bfeebac4cc9ad851cd4a6a7d3b0 for earlier term 5. Current term is 6. Ops: []
I20250905 20:00:32.729475 31463 raft_consensus.cc:513] T a042aed9ffcb49c19e07c53c69bd1b95 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:32.803988 31471 leader_election.cc:290] T 0adebfdecbae47298f234b849fcea7fc P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 10 election: Requested vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
W20250905 20:00:32.814358 31339 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.755s user 0.006s sys 0.070s
I20250905 20:00:32.851768 28155 raft_consensus.cc:1273] T 7ca47360fb0f4be6a95a798d0a6d25b8 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 11 FOLLOWER]: Refusing update from remote peer cd764bfeebac4cc9ad851cd4a6a7d3b0: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20250905 20:00:32.794725 31358 raft_consensus.cc:695] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 LEADER]: Becoming Leader. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:32.802137 31321 raft_consensus.cc:491] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:32.819257 31426 raft_consensus.cc:3058] T e707b0f561934674947fb463ce626a07 P e4413ae5e0b64b52a6f360ac580239b9 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:32.853945 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0ac2dfe59d423189dd8d72eee98a42" candidate_uuid: "7e9c9f591f05464f9b097bf003d14f4a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" is_pre_election: true
I20250905 20:00:32.857975 31220 raft_consensus.cc:3058] T 2f356a2522e646889c023d4b3d26f9cc P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:32.861598 28083 raft_consensus.cc:2466] T 4414c08fe910403b80af2d558aac36b2 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:32.850701 31216 raft_consensus.cc:2747] T 9a9976ebf1c1415a85592e195bad9f36 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250905 20:00:32.870750 28717 leader_election.cc:304] T 0adebfdecbae47298f234b849fcea7fc P 7e9c9f591f05464f9b097bf003d14f4a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, 7e9c9f591f05464f9b097bf003d14f4a; no voters:
I20250905 20:00:32.854650 31282 raft_consensus.cc:3058] T a3efc6666ba5411c9b50c9d6e34089ad P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:32.855355 31171 raft_consensus.cc:738] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cd764bfeebac4cc9ad851cd4a6a7d3b0, State: Running, Role: LEADER
I20250905 20:00:32.918303 31171 consensus_queue.cc:260] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:32.920038 31171 raft_consensus.cc:3058] T 8afd77a8d80843aa805f971a2806ab86 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 7 FOLLOWER]: Advancing to term 8
I20250905 20:00:32.860940 28050 leader_election.cc:304] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b2f874b8e54f08b01a9ed00d5cce05, e4413ae5e0b64b52a6f360ac580239b9; no voters:
W20250905 20:00:32.923453 28050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.58.131:36705, user_credentials={real_user=slave}} blocked reactor thread for 62589.5us
W20250905 20:00:32.923791 31500 log.cc:927] Time spent T 5c21c3755f90405db03f98780efdb0b5 P 26b2f874b8e54f08b01a9ed00d5cce05: Append to log took a long time: real 0.068s user 0.000s sys 0.001s
I20250905 20:00:32.924893 31439 raft_consensus.cc:2802] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250905 20:00:32.925334 31439 raft_consensus.cc:491] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 20:00:32.925937 31439 raft_consensus.cc:3058] T ef7f42a26b64466a9d6b1eae6df0e614 P e4413ae5e0b64b52a6f360ac580239b9 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:32.925230 31358 consensus_queue.cc:237] T e802ce1da002477b8896acf7bf7c84a7 P cd764bfeebac4cc9ad851cd4a6a7d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:32.931010 28150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0adebfdecbae47298f234b849fcea7fc" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:32.931820 28150 raft_consensus.cc:3058] T 0adebfdecbae47298f234b849fcea7fc P 26b2f874b8e54f08b01a9ed00d5cce05 [term 8 FOLLOWER]: Advancing to term 10
I20250905 20:00:32.939973 28157 raft_consensus.cc:604] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Received request to transfer leadership to TS cd764bfeebac4cc9ad851cd4a6a7d3b0
I20250905 20:00:32.940557 28157 raft_consensus.cc:612] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250905 20:00:32.869256 31341 raft_consensus.cc:3058] T 10df9d019f2e4e208f4e23d50f551ad6 P 7e9c9f591f05464f9b097bf003d14f4a [term 8 FOLLOWER]: Advancing to term 9
W20250905 20:00:32.943001 28124 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:32.943015 28159 raft_consensus.cc:1273] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Refusing update from remote peer e4413ae5e0b64b52a6f360ac580239b9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20250905 20:00:32.944253 28159 pending_rounds.cc:85] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting all ops after (but not including) 3
I20250905 20:00:32.944501 28159 pending_rounds.cc:107] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20250905 20:00:32.944741 28159 raft_consensus.cc:2887] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250905 20:00:32.954330 31212 raft_consensus.cc:491] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:32.954953 31212 raft_consensus.cc:513] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "e4413ae5e0b64b52a6f360ac580239b9" member_type: VOTER last_known_addr { host: "127.26.58.130" port: 34193 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } }
I20250905 20:00:32.955955 28159 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33756 (request call id 2361) took 1071 ms. Trace:
I20250905 20:00:32.956354 28159 rpcz_store.cc:276] 0905 20:00:31.884261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:31.925657 (+ 41396us) service_pool.cc:225] Handling call
0905 20:00:31.973506 (+ 47849us) raft_consensus.cc:1469] Updating replica for 0 ops
0905 20:00:32.945592 (+972086us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 20:00:32.955654 (+ 10062us) spinlock_profiling.cc:243] Waited 969 ms on lock 0x7b54000d1954. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc19dc1 00007fe49dc187fe 00007fe4a2b7cc99 00007fe49c202b44 00007fe49c202ad2 00007fe49c202a01 00007fe49c20297c 00007fe49c201c32 00007fe49c018ddc 00007fe49c018216
0905 20:00:32.955844 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2227979008}
I20250905 20:00:32.957077 31212 leader_election.cc:290] T 0e4ec392d0d54040b0b2162eb0601c0d P e4413ae5e0b64b52a6f360ac580239b9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cd764bfeebac4cc9ad851cd4a6a7d3b0 (127.26.58.129:38421), 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705)
I20250905 20:00:32.958004 28157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:36164 (request call id 30) took 1526 ms. Trace:
I20250905 20:00:32.882367 31321 raft_consensus.cc:513] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd764bfeebac4cc9ad851cd4a6a7d3b0" member_type: VOTER last_known_addr { host: "127.26.58.129" port: 38421 } } peers { permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05" member_type: VOTER last_known_addr { host: "127.26.58.131" port: 36705 } } peers { permanent_uuid: "7e9c9f591f05464f9b097bf003d14f4a" member_type: VOTER last_known_addr { host: "127.26.58.132" port: 44299 } attrs { promote: false } }
I20250905 20:00:32.885388 31427 raft_consensus.cc:3053] T 9a9976ebf1c1415a85592e195bad9f36 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 7 LEADER]: Stepping down as leader of term 7
I20250905 20:00:32.894326 28085 raft_consensus.cc:1238] T 4f2279535926437ba84ae91f2b83b3d3 P e4413ae5e0b64b52a6f360ac580239b9 [term 8 FOLLOWER]: Rejecting Update request from peer 26b2f874b8e54f08b01a9ed00d5cce05 for earlier term 7. Current term is 8. Ops: []
W20250905 20:00:32.899701 31499 log.cc:927] Time spent T fc7dc431504743d79dbdc8eea83f9dcb P e4413ae5e0b64b52a6f360ac580239b9: Append to log took a long time: real 0.180s user 0.002s sys 0.000s
I20250905 20:00:32.856505 31453 raft_consensus.cc:3058] T 1d1a8f2db2604a879529dda40b457a4a P cd764bfeebac4cc9ad851cd4a6a7d3b0 [term 5 FOLLOWER]: Advancing to term 6
I20250905 20:00:32.958578 31315 raft_consensus.cc:491] T 5f31007f29044bd1817fcf19ba904ad3 P 26b2f874b8e54f08b01a9ed00d5cce05 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 20:00:32.872118 28078 raft_consensus.cc:2466] T 1f65256308814d3a9a4a01db1b910611 P e4413ae5e0b64b52a6f360ac580239b9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 6.
I20250905 20:00:32.975999 28748 raft_consensus.cc:2466] T f2de38aacf1b4dccaeee999400fb6d75 P 7e9c9f591f05464f9b097bf003d14f4a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b2f874b8e54f08b01a9ed00d5cce05 in term 3.
I20250905 20:00:32.959898 28157 rpcz_store.cc:276] 0905 20:00:31.432073 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 20:00:31.432281 (+ 208us) service_pool.cc:225] Handling call
0905 20:00:32.957768 (+1525487us) spinlock_profiling.cc:243] Waited 1.5 s on lock 0x7b54000d1954. stack: 00007fe49ac72473 00007fe49ac72370 00007fe49a5c8574 00007fe4a40e8784 00007fe4a40e8719 00007fe49db9d6e1 00007fe49dc13813 00007fe4a2b801ca 00007fe49c2131c4 00007fe49c213152 00007fe49c213081 00007fe49c212ffc 00007fe49c2122b2 00007fe49c018ddc 00007fe49c018216 00007fe49c017bbf
0905 20:00:32.957909 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3457316736}
W20250905 20:00:32.980476 28048 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 20:00:33.000205 28159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f2279535926437ba84ae91f2b83b3d3" candidate_uuid: "e4413ae5e0b64b52a6f360ac580239b9" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"
I20250905 20:00:33.005144 31321 leader_election.cc:290] T c0889204295e4daa99d81e5b2a45ac1c P cd764bfeebac4cc9ad851cd4a6a7d3b0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 26b2f874b8e54f08b01a9ed00d5cce05 (127.26.58.131:36705), 7e9c9f591f05464f9b097bf003d14f4a (127.26.58.132:44299)
I20250905 20:00:32.867312 31457 consensus_queue.cc:1035] T a042aed9ffcb49c19e07c53c69bd1b95 P e4413ae5e0b64b52a6f360ac580239b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b2f874b8e54f08b01a9ed00d5cce05"